CN109800223A - Log processing method, device, electronic equipment and storage medium - Google Patents
Log processing method, device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN109800223A CN109800223A CN201811520640.XA CN201811520640A CN109800223A CN 109800223 A CN109800223 A CN 109800223A CN 201811520640 A CN201811520640 A CN 201811520640A CN 109800223 A CN109800223 A CN 109800223A
- Authority
- CN
- China
- Prior art keywords
- log
- index
- processing method
- query
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The embodiment provides a kind of log processing method, device, electronic equipment and storage mediums, are related to field of computer technology.This method comprises: being handled by unified journal format specification the log of multiple operation systems；The log of the multiple operation system is matched by regular expression, obtains the information of the journal format specification of the log, the information of the journal format specification includes log generation time, logging level, operation system mark and server identification；The information of the journal format specification based on the log establishes the index of the log；Index based on the log stores the log to target database.The technical solution of the embodiment of the present invention can make the log of each operation system unitized, be convenient for batch processing log, can be improved the search efficiency of log.
Description
Technical field
The present invention relates to field of computer technology, in particular to a kind of log processing method, log processing device,
Electronic equipment and computer readable storage medium.
Background technique
With the development of internet technology, the user of each operation system is more and more, with the increase of user volume, business
System produces the business diary of magnanimity, how to be efficiently processed into business diary for focus of attention.
In existing technical solution, there are many quantity of the server of each operation system, but operation system is not
Unified log platform, the ununified specification of the business diary that operation system generates, will take one by one in inquiry log
Business device is handled, it is difficult to carry out batch processing and inquiry.
Therefore, how efficiently to carry out batch log processing becomes technical problem urgently to be resolved.
It should be noted that information is only used for reinforcing the reason to background of the present invention disclosed in above-mentioned background technology part
Solution, therefore may include the information not constituted to the prior art known to persons of ordinary skill in the art.
Summary of the invention
The embodiment of the present invention be designed to provide a kind of log processing method, log processing device, electronic equipment and
Computer readable storage medium, and then overcome one caused by the limitation and defect due to the relevant technologies at least to a certain extent
A or multiple problems.
According to a first aspect of the embodiments of the present invention, a kind of log processing method is provided, comprising: pass through unified log
Format specification handles the log of multiple operation systems；By regular expression to the log of the multiple operation system into
Row matching obtains the information of the journal format specification of the log, when the information of the journal format specification includes that log generates
Between, logging level, operation system mark and server identification；The information of the journal format specification based on the log is built
Found the index of the log；Index based on the log stores the log to target database.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the journal format based on the log
The information of specification establishes the index of the log, comprising: chooses institute from the information of the journal format specification of the log
State one or more fields in log generation time, logging level, operation system mark and server identification；Based on selection
One or more fields establish the log one or more fields index.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing method further include: connecing
When receiving log query request, one or more logs mark of the log is extracted from log query request；It is based on
The quantity of one or more log marks determines the index word number of segment used；Based on the index word number of segment and the day
The index of will inquires corresponding log from the target database.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing method further include: connecing
When receiving log aggregation inquiry request, log query is carried out by multiple servers of the shell script to the operation system；It is right
The query result of every server carries out aggregate query, carries out secondary analysis with the query result to each every server.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing method further include: from going through
The sentence inquired and parameters are extracted in the request of history log query, the sentence inquired and parameters are saved as
Historical query record；When receiving log query request, the log query is asked based on historical query record
It asks and is handled.
In some exemplary embodiments of the invention, aforementioned schemes are based on, the target database is HBASE data
Library, the index of the log are that the row of the HBASE database is strong.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing method further include: connecing
When receiving log query request, log analysis query template corresponding with log query request is obtained；Based on the log
Analyze query template and index inquiry log from the target database.
According to a second aspect of the embodiments of the present invention, a kind of log processing device is provided, comprising: unit is uniformly processed, uses
The log of multiple operation systems is handled in by unified journal format specification；Information acquisition unit, for by just
Then expression formula matches the log of the multiple operation system, obtains the information of the journal format specification of the log, institute
The information for stating journal format specification includes log generation time, logging level, operation system mark and server identification；Index
Unit is established, the information for the journal format specification based on the log establishes the index of the log；Storage unit,
The log is stored to target database for the index based on the log.
According to a third aspect of the embodiments of the present invention, a kind of electronic equipment is provided, comprising: processor；And memory, institute
It states and is stored with computer-readable instruction on memory, realize when the computer-readable instruction is executed by the processor as above-mentioned
Log processing method described in first aspect.
According to a third aspect of the embodiments of the present invention, a kind of computer readable storage medium is provided, calculating is stored thereon with
Machine program realizes the log processing method as described in above-mentioned first aspect when the computer program is executed by processor.
In the technical solution provided by some embodiments of the present invention, on the one hand, pass through unified journal format specification
The log of operation system is handled, the log of each operation system can be made unitized, be convenient for batch processing log；It is another
The information of aspect, the journal format specification based on log establishes the index of log, is stored log to mesh based on the index of log
It marks in database, convenient for can be improved the search efficiency of log based on index batch query log.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not
It can the limitation present invention.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows and meets implementation of the invention
Example, and be used to explain the principle of the present invention together with specification.It should be evident that the accompanying drawings in the following description is only the present invention
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.In the accompanying drawings:
Fig. 1 shows the flow diagram of log inquiring method according to some embodiments of the invention；
Fig. 2 shows the flow diagrams for the index for establishing log according to some embodiments of the present invention；
Fig. 3 shows the flow diagram for carrying out log query according to some embodiments of the present invention；
Fig. 4 shows the schematic block diagram of the log query device of an exemplary embodiment according to the present invention；
Fig. 5 shows the structural schematic diagram for being suitable for the computer system for the electronic equipment for being used to realize the embodiment of the present invention.
Specific embodiment
Example embodiment is described more fully with reference to the drawings.However, example embodiment can be real in a variety of forms
It applies, and is not understood as limited to embodiment set forth herein；On the contrary, thesing embodiments are provided so that the present invention will be comprehensively and complete
It is whole, and the design of example embodiment is comprehensively communicated to those skilled in the art.Identical appended drawing reference indicates in figure
Same or similar part, thus repetition thereof will be omitted.
In addition, described feature, structure or characteristic can be incorporated in one or more implementations in any suitable manner
In example.In the following description, many details are provided to provide and fully understand to the embodiment of the present invention.However,
It will be appreciated by persons skilled in the art that technical solution of the present invention can be practiced without one or more in specific detail,
Or it can be using other methods, constituent element, device, step etc..In other cases, it is not shown in detail or describes known side
Method, device, realization or operation are to avoid fuzzy each aspect of the present invention.
Block diagram shown in the drawings is only functional entity, not necessarily must be corresponding with physically separate entity.
I.e., it is possible to realize these functional entitys using software form, or realized in one or more hardware modules or integrated circuit
These functional entitys, or these functional entitys are realized in heterogeneous networks and/or processor device and/or microcontroller device.
Flow chart shown in the drawings is merely illustrative, it is not necessary to including all content and operation/step,
It is not required to execute by described sequence.For example, some operation/steps can also decompose, and some operation/steps can close
And or part merge, therefore the sequence actually executed is possible to change according to the actual situation.
Fig. 1 shows the flow diagram of log inquiring method according to some embodiments of the invention, the log query side
Method is applied to server end.Shown in referring to Fig.1, which may include step S110 to step S140.Next,
Log inquiring method in example embodiment to Fig. 1 is described in detail.
Shown in referring to Fig.1, in step s 110, by unified journal format specification to the logs of multiple operation systems into
Row processing.
In the exemplary embodiment, multiple operation systems may include: insurance business system, stock exchange transaction system, financial circles
Business system.In addition, operation system can also include subservice system, such as stock exchange transaction platform may include scalp subsystem,
Reconciliation subsystem and clearance subsystem etc..
In the exemplary embodiment, journal format specification includes: time: log generation time, ISO8601 format；Level:
Logging level, FATAL, ERROR, WARN, INFO, DEBUG；Serial_id: business id, for which industry of log source to be indicated
Business system；Server_id: server id, which platform server of operation system is derived from for distinguishing log.Except above-mentioned field
Except, business side can also voluntarily add additional field, such as APP_id, using id, for distinguishing log from which
Application program.
In the step s 120, the log of the multiple operation system is matched by regular expression, described in acquisition
The information of the journal format specification of log, the information of the journal format specification include log generation time, logging level, business
System banner and server identification.
In the exemplary embodiment, carry out cutting log using the mode of text fragments cutting, by normal expression formula to multiple
The log of operation system is matched, and the information of the journal format specification of the log of each operation system is obtained.Regular expression
Basic syntax it is as follows: % { SYNTAX:SEMANTIC } * ' SYNTAX ' represents the type of matching value, for example, " 0.11 " can be with
" NUMBER " type is matched, and " 10.222.22.25 " can be with " IP " type matching.Wherein, " SEMANTIC " indicates that storage should
One variable declarations of value, for doing field searches and statistics.
In the exemplary embodiment, the regular expression library of log processing is established, is stored in regular expression library multiple
Regular expression template.Can by the expression formula in the regular expression template in built-in regular expression library to log into
Row matching, such as by timestamp expression { TIMESTAMP_ISO8601:timestamp } come match log generation time when
Between stab.It can be matched with customized regular expression, for example, customized canonical tabular form % { YEAR } [/ -] %
{ MONTHNUM } [/ -] % { MONTHDAY } matches the timestamp in log.The following are the days extracted by regular expression
Will generation time, logging level, operation system mark, the example of server identification: { " time ": " 2017-08-04,15:59:
01 ", " level ": " INFO ", " Service_id ": " insurance business 1 ", " Server_id ": " Server1 ", " log_id ": "
Log1 ", " log ": " trading successfully " }
In step s 130, the information of the journal format specification based on the log establishes the index of the log.
In the exemplary embodiment, if log generation time, logging level, operation system mark, server identification are followed successively by
A, tetra- fields of B, C, D.When establishing the index of log, it can establish four field index, three field index, two field ropes of log
Draw.By taking four field index as an example, four field index are tetra- field of ABCD, which supports that ABCD, ABC, AB and A are total
Four kinds of inquiries, with A, B, C, D beginning, circulation takes fully three fields to the index of three fields backward respectively, obtains: (it will by ABC, BCD
Support tri- kinds of BCD, BC and B inquiries simultaneously), (it will be propped up simultaneously by CDA (it will support simultaneously tri- kinds of CDA, CD and C inquiry) and DAB
Hold tri- kinds of DAB, DA and D inquiries), two field index will be respectively from three three field index remained successively with each
Field beginning takes fully two fields, then removal repeat and prefix overlapping index, finally obtain DB (it will support simultaneously DB and
Two kinds of D inquiries) and AC (it will support AC and two kinds of A inquiries simultaneously).
In step S140, the index based on the log stores the log to target database.
In the exemplary embodiment, target database can be HBASE database, and the index of log can be HBASE data
The line unit in library.The field information corresponding with index that each log is extracted by regular expression, corresponding field information is deposited
The log is stored in target database position corresponding with the index under corresponding index field by storage.
According to the log processing method in the example embodiment of Fig. 1, on the one hand, by unified journal format specification to industry
The log of business system is handled, and the log of each operation system can be made unitized, be convenient for batch processing log；Another party
The information in face, the journal format specification based on log establishes the index of log, is stored log to target based on the index of log
In database, convenient for can be improved the search efficiency of log based on index batch query log.
Further, in the exemplary embodiment, it when receiving log query request, obtains and is requested with the log query
Corresponding log analysis query template；Based on the log analysis query template and the index from the target database
Inquiry log.In the exemplary embodiment, common log analysis query template is provided, user can select usage log point on demand
Query template is analysed, to realize complicated log analysis function.
In addition, in the exemplary embodiment, the sentence inquired and corresponding parameters are saved as historical query record；
Log query request is handled based on historical query record.The inquiry each time of user has one record of generation,
Sentence that user query are crossed, all parameters, query result can all be automatically saved, there is function of search in query page, it can basis
The inquiry of conditional search history records and as a result, if it is desired to executing again, it is only necessary to click and execute, for the first time without input
The information such as IP for needing to input when inquiry, the information such as query script.In addition, query page is also supported in original querying condition
On the basis of modify every querying condition and inquired again, flexibly and easily.
Further, in the exemplary embodiment, when receiving log aggregation inquiry request, by shell script to described
The multiple servers of operation system carry out log query；Aggregate query is carried out to the query result of every server, to each
The query result of every server carries out secondary analysis.User fill in this inquiry title, server ip list, shell script,
User name, syndication option, polymerization processing script, then submit query analysis, and shell script is committed to IP column simultaneously by this function
It is concurrently executed on the specified Servers-all of table, implementing result will carry out after polymerization script to result after returning, finally
As a result it is put in storage and is sent directly to the mailbox of user, consults log analysis result for user.Different query function customizables
Different titles, IP refer to the server ip list that needs are inquired, and script refers to that executable shell script (supports canonical table
Up to formula) etc., such as: grep " keys "/tmp/test.log such as counts the number of some url access, needs if necessary to polymerize
Access times on Servers-all are added, needs to fill in polymerization script in polymerization processing column, for above example, gather
Fit originally is to be added the access times of each server url.This batch query tool provides common log analysis inquiry
Template, user can be selected on demand using the log analysis function that complexity can be realized.
It should be noted that can be periodically to operation system execution journal aggregate query, for example, can be with day, the moon
Or season is to carry out aggregate query to the log stored on the multiple servers of operation system in the period, can also needed to business
Automatic execution journal aggregate query when the log of system is analyzed, this is same within the scope of the present invention.Fig. 2 shows
The flow diagram of the index of log is established according to some embodiments of the present invention.
Referring to shown in Fig. 2, in step S210, from the information of the journal format specification of the log described in selection
One or more fields in log generation time, logging level, operation system mark and server identification.
In the exemplary embodiment, the log generation time, logging level, business system can be extracted by regular expression
One of system mark, server instance mark or a variety of logs mark.
In step S220, one or more fields based on selection establish the rope of one or more fields of the log
Draw.
In the exemplary embodiment, four field index, three field index, two field index of log be can establish, with four words
For segment index, four field index are tetra- field of ABCD, which supports ABCD, ABC, AB and A totally four kinds of inquiries,
With A, B, C, D beginning, circulation takes fully three fields to the index of three fields backward respectively, and obtain: (it will be supported simultaneously by ABC, BCD
Tri- kinds of BCD, BC and B inquiries), (it will support DAB, DA simultaneously by CDA (it will support simultaneously tri- kinds of CDA, CD and C inquiry) and DAB
Inquired with tri- kinds of D), two field index will be respectively from three three field index remained successively with the beginning of each field
Two fields are taken fully, then removal repeats and the index of prefix overlapping, and finally obtaining DB, (it will support DB simultaneously and two kinds of D are looked into
Ask) and AC (it will support AC and two kinds of A inquiries simultaneously).
Fig. 3 shows the flow diagram for carrying out log query according to some embodiments of the present invention.
Referring to shown in Fig. 3, in step s310, when receiving log query request, from log query request
Extract one or more logs mark of the log.
In the exemplary embodiment, when receiving log query request, the log can be extracted by regular expression
One of generation time, logging level, operation system mark, server instance mark or a variety of logs mark.
In step s 320, the index word number of segment used is determined based on the quantity of one or more log marks.
In the exemplary embodiment, the index word number of segment used is determined based on the quantity of one or more log marks.
For example, carrying out log query using two field index when the quantity of log mark is two.
In step S330, the index based on the index word number of segment and the log is looked into from the target database
Ask corresponding log.
In the exemplary embodiment, based on determining the index of the index word number of segment and the log that use from the number of targets
According to inquiring corresponding log in library.For example, three field index such as log generation time, logging level level-one business can be passed through
System banner inquires the log of the warning registration of the insurance business system of certain a period of time.Log is carried out based on the index of log
Inquiry, can be realized batch log query, improves log query efficiency.
Further, index is established to the logged result analyzed, is based on log generation time, logging level, business system
System mark, server identification establish respective index, and target database can be HBASE database, and the index of log can be
The line unit of HBASE database.Index based on log inquires log, can be realized batch log query, improves day
Will search efficiency.Search index based on logged result can quickly and efficiently carry out effect of visualization displaying, for example show service
The log amount of device dimension, the log distribution of operation system maintenance, average response time etc..
It can be in addition, in some embodiments it is possible to customize the query script of each server performance index, after script is fixed
Set time, fixed frequency carry out detection script execution to the Servers-all of cluster environment as a result, using result as distribution
The performance indicator of server in cluster can trigger mail alarm for abnormal index, the achievement data of collection can be unified the page
It shows.
In addition, in an embodiment of the present invention, additionally providing a kind of log processing device.Referring to shown in Fig. 4, at the log
Reason device may include: that unit 410, information acquisition unit 420, index is uniformly processed to establish unit 430 and storage unit
440.Wherein, be uniformly processed unit 410 for by unified journal format specification to the log of multiple operation systems at
Reason；Information acquisition unit 420 is for matching the log of the multiple operation system by regular expression, described in acquisition
The information of the journal format specification of log, the information of the journal format specification include log generation time, logging level, business
System banner and server identification；Index establishes letter of the unit 430 for the journal format specification based on the log
Breath establishes the index of the log；Storage unit 440 stores the log to number of targets for the index based on the log
According to library.
In some exemplary embodiments of the invention, aforementioned schemes are based on, it includes: field choosing that index, which establishes unit 430,
Take unit, for chosen from the information of the journal format specification of the log log generation time, logging level,
One or more fields in operation system mark and server identification；Unit is established, for one or more based on selection
A field establishes the index of one or more fields of the log.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing device 400 further include:
Marker extraction unit, for extracting the one of the log from log query request when receiving log query request
Kind or a variety of logs mark；Field Count determination unit is used for being determined based on the quantity of one or more log marks
Index word number of segment；Query unit, for the index word number of segment and the index of the log from the target database
Inquire corresponding log.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing device 400 further include:
Script query unit, for when receiving log aggregation inquiry request, by shell script to more of the operation system
Server carries out log query；Aggregate query unit carries out aggregate query for the query result to every server, to every
The query result of a every server carries out secondary analysis.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing device 400 further include:
Storage unit is recorded, for extracting the sentence inquired and parameters from history log inquiry request, was inquired described
Sentence and parameters save as historical query record；Processing unit, for receive the log query request when, base
Log query request is handled in historical query record.
In some exemplary embodiments of the invention, aforementioned schemes are based on, the target database is HBASE data
Library, the index of the log are that the row of the HBASE database is strong.
In some exemplary embodiments of the invention, it is based on aforementioned schemes, the log processing device 400 further include:
Template acquiring unit, for obtaining log analysis corresponding with log query request when receiving log query request
Query template；Query unit, for being based on the log analysis query template and the index from the target database
Inquiry log.
Each functional module and above-mentioned log processing side due to the log processing device 400 of example embodiments of the present invention
The step of example embodiment of method, is corresponding, therefore details are not described herein.
In an exemplary embodiment of the present invention, a kind of electronic equipment that can be realized the above method is additionally provided.
Below with reference to Fig. 5, it illustrates the computer systems 500 for the electronic equipment for being suitable for being used to realize the embodiment of the present invention
Structural schematic diagram.The computer system 500 of electronic equipment shown in Fig. 5 is only an example, should not be to the embodiment of the present invention
Function and use scope bring any restrictions.
As shown in figure 5, computer system 500 includes central processing unit (CPU) 501, it can be read-only according to being stored in
Program in memory (ROM) 502 or be loaded into the program in random access storage device (RAM) 503 from storage section 508 and
Execute various movements appropriate and processing.In RAM 503, it is also stored with various programs and data needed for system operatio.CPU
501, ROM 502 and RAM 503 is connected with each other by bus 504.Input/output (I/O) interface 505 is also connected to bus
504。
I/O interface 505 is connected to lower component: the importation 506 including keyboard, mouse etc.；It is penetrated including such as cathode
The output par, c 507 of spool (CRT), liquid crystal display (LCD) etc. and loudspeaker etc.；Storage section 508 including hard disk etc.；
And the communications portion 509 of the network interface card including LAN card, modem etc..Communications portion 509 via such as because
The network of spy's net executes communication process.Driver 510 is also connected to I/O interface 505 as needed.Detachable media 511, such as
Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on as needed on driver 510, in order to read from thereon
Computer program be mounted into storage section 508 as needed.
Particularly, according to an embodiment of the invention, may be implemented as computer above with reference to the process of flow chart description
Software program.For example, the embodiment of the present invention includes a kind of computer program product comprising be carried on computer-readable medium
On computer program, which includes the program code for method shown in execution flow chart.In such reality
It applies in example, which can be downloaded and installed from network by communications portion 509, and/or from detachable media
511 are mounted.When the computer program is executed by central processing unit (CPU) 501, executes and limited in the system of the application
Above-mentioned function.
It should be noted that computer-readable medium shown in the present invention can be computer-readable signal media or meter
Calculation machine readable storage medium storing program for executing either the two any combination.Computer readable storage medium for example can be --- but not
Be limited to --- electricity, magnetic, optical, electromagnetic, infrared ray or semiconductor system, device or device, or any above combination.Meter
The more specific example of calculation machine readable storage medium storing program for executing can include but is not limited to: have the electrical connection, just of one or more conducting wires
Taking formula computer disk, hard disk, random access storage device (RAM), read-only memory (ROM), erasable type may be programmed read-only storage
Device (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device,
Or above-mentioned any appropriate combination.In the present invention, computer readable storage medium can be it is any include or storage journey
The tangible medium of sequence, the program can be commanded execution system, device or device use or in connection.And at this
In invention, computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,
Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimited
In electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer can
Any computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used for
By the use of instruction execution system, device or device or program in connection.Include on computer-readable medium
Program code can transmit with any suitable medium, including but not limited to: wireless, electric wire, optical cable, RF etc. are above-mentioned
Any appropriate combination.
Flow chart and block diagram in attached drawing are illustrated according to the system of various embodiments of the invention, method and computer journey
The architecture, function and operation in the cards of sequence product.In this regard, each box in flowchart or block diagram can generation
A part of one module, program segment or code of table, a part of above-mentioned module, program segment or code include one or more
Executable instruction for implementing the specified logical function.It should also be noted that in some implementations as replacements, institute in box
The function of mark can also occur in a different order than that indicated in the drawings.For example, two boxes succeedingly indicated are practical
On can be basically executed in parallel, they can also be executed in the opposite order sometimes, and this depends on the function involved.Also it wants
It is noted that the combination of each box in block diagram or flow chart and the box in block diagram or flow chart, can use and execute rule
The dedicated hardware based systems of fixed functions or operations is realized, or can use the group of specialized hardware and computer instruction
It closes to realize.
Being described in unit involved in the embodiment of the present invention can be realized by way of software, can also be by hard
The mode of part realizes that described unit also can be set in the processor.Wherein, the title of these units is in certain situation
Under do not constitute restriction to the unit itself.
As on the other hand, present invention also provides a kind of computer-readable medium, which be can be
Included in electronic equipment described in above-described embodiment；It is also possible to individualism, and without in the supplying electronic equipment.
Above-mentioned computer-readable medium carries one or more program, when the electronics is set by one for said one or multiple programs
When standby execution, so that the electronic equipment realizes such as above-mentioned log processing method as described in the examples.
For example, the electronic equipment may be implemented as shown in Figure 1: step S110 is advised by unified journal format
Model handles the log of multiple operation systems；Step S120, by regular expression to the day of the multiple operation system
Will is matched, and the information of the journal format specification of the log is obtained, and the information of the journal format specification includes that log produces
Raw time, logging level, operation system mark and server identification；Step S130, the log lattice based on the log
The information of formula specification establishes the index of the log；Step S140, the index based on the log store the log to mesh
Mark database.
It should be noted that although being referred to several modules for acting the device executed in the above detailed description
Or unit, but this division is not enforceable.In fact, embodiment according to the present invention, above-described two
Or more the feature and function of module or unit can be embodied in a module or unit.Conversely, above-described
One module or the feature and function of unit can be to be embodied by multiple modules or unit with further division.
Through the above description of the embodiments, those skilled in the art is it can be readily appreciated that example described herein is implemented
Mode can also be realized by software realization in such a way that software is in conjunction with necessary hardware.Therefore, according to the present invention
The technical solution of embodiment can be embodied in the form of software products, which can store non-volatile at one
Property storage medium (can be CD-ROM, USB flash disk, mobile hard disk etc.) in or network on, including some instructions are so that a calculating
Equipment (can be personal computer, server, touch control terminal or network equipment etc.) executes embodiment according to the present invention
Method.
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to of the invention its
Its embodiment.This application is intended to cover any variations, uses, or adaptations of the invention, these modifications, purposes or
Person's adaptive change follows general principle of the invention and including the undocumented common knowledge in the art of the present invention
Or conventional techniques.The description and examples are only to be considered as illustrative, and true scope and spirit of the invention are by following
Claim is pointed out.
It should be understood that the present invention is not limited to the precise structure already described above and shown in the accompanying drawings, and
And various modifications and changes may be made without departing from the scope thereof.The scope of the present invention is limited only by the attached claims.
Claims (10)
1. a kind of log processing method characterized by comprising
The log of multiple operation systems is handled by unified journal format specification；
The log of the multiple operation system is matched by regular expression, obtains the journal format specification of the log
Information, the information of the journal format specification include log generation time, logging level, operation system mark and server
Mark；
The information of the journal format specification based on the log establishes the index of the log；
Index based on the log stores the log to target database.
2. log processing method according to claim 1, which is characterized in that the journal format rule based on the log
The information of model establishes the index of the log, comprising:
The log generation time, logging level, business system are chosen from the information of the journal format specification of the log
One or more fields in system mark and server identification；
One or more fields based on selection establish the index of one or more fields of the log.
3. log processing method according to claim 1, which is characterized in that the log processing method further include:
When receiving log query request, one or more log marks of the log are extracted from log query request
Know；
The index word number of segment used is determined based on the quantity of one or more log marks；
Index based on the index word number of segment and the log inquires corresponding log from the target database.
4. log processing method according to claim 1, which is characterized in that the log processing method further include:
When receiving log aggregation inquiry request, log is carried out by multiple servers of the shell script to the operation system
Inquiry；
Aggregate query is carried out to the query result of every server, carries out secondary point with the query result to each every server
Analysis.
5. log processing method according to claim 3, which is characterized in that the log processing method further include:
The sentence inquired and parameters are extracted from history log inquiry request, by the sentence inquired and every ginseng
Number saves as historical query record；
When receiving log query request, recorded based on the historical query at log query request
Reason.
6. log processing method according to any one of claim 1 to 5, which is characterized in that the target database is
HBASE database, the index of the log are that the row of the HBASE database is strong.
7. log processing method according to claim 1 or 2, which is characterized in that the log processing method further include:
When receiving log query request, log analysis query template corresponding with log query request is obtained；
Based on the log analysis query template and index inquiry log from the target database.
8. a kind of log processing device characterized by comprising
Unit is uniformly processed, for handling by unified journal format specification the log of multiple operation systems；
Information acquisition unit, for being matched by regular expression to the log of the multiple operation system, described in acquisition
The information of the journal format specification of log, the information of the journal format specification include log generation time, logging level, business
System banner and server identification；
Index establishes unit, and the information for the journal format specification based on the log establishes the index of the log；
Storage unit stores the log to target database for the index based on the log.
9. a kind of electronic equipment characterized by comprising
Processor；And
Memory is stored with computer-readable instruction on the memory, and the computer-readable instruction is held by the processor
The log processing method as described in any one of claims 1 to 7 is realized when row.
10. a kind of computer readable storage medium, is stored thereon with computer program, the computer program is executed by processor
Log processing method of the Shi Shixian as described in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811520640.XA CN109800223A (en) | 2018-12-12 | 2018-12-12 | Log processing method, device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811520640.XA CN109800223A (en) | 2018-12-12 | 2018-12-12 | Log processing method, device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109800223A true CN109800223A (en) | 2019-05-24 |
Family
ID=66556662
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811520640.XA Pending CN109800223A (en) | 2018-12-12 | 2018-12-12 | Log processing method, device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109800223A (en) |
Cited By (35)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110569214A (en) * | 2019-08-02 | 2019-12-13 | 杭州云纪网络科技有限公司 | Index construction method and device for log file and electronic equipment |
| CN110674159A (en) * | 2019-09-02 | 2020-01-10 | 平安科技（深圳）有限公司 | Service query method and related device |
| CN110716910A (en) * | 2019-10-14 | 2020-01-21 | 中国建设银行股份有限公司 | Log management method, device, equipment and storage medium |
| CN110727641A (en) * | 2019-10-21 | 2020-01-24 | 中国民航信息网络股份有限公司 | Log searching method and device |
| CN110730109A (en) * | 2019-10-12 | 2020-01-24 | 北京百度网讯科技有限公司 | Method and apparatus for generating information |
| CN110889068A (en) * | 2019-10-23 | 2020-03-17 | 北京达佳互联信息技术有限公司 | Method and device for assisting in consulting log and electronic equipment |
| CN111046010A (en) * | 2019-11-13 | 2020-04-21 | 泰康保险集团股份有限公司 | Log storage method, device, system, electronic equipment and computer readable medium |
| CN111047427A (en) * | 2019-11-26 | 2020-04-21 | 深圳市卡牛科技有限公司 | Data reporting method, device, server and storage medium |
| CN111144086A (en) * | 2019-12-20 | 2020-05-12 | 锐捷网络股份有限公司 | Log formatting method and device, electronic equipment and storage medium |
| CN111176762A (en) * | 2019-12-31 | 2020-05-19 | 北京奇艺世纪科技有限公司 | Batch processing program execution method and device, electronic equipment and storable medium |
| CN111241137A (en) * | 2020-01-08 | 2020-06-05 | 北京字节跳动网络技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111324510A (en) * | 2020-02-21 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Log processing method and device and electronic equipment |
| CN111522714A (en) * | 2020-04-20 | 2020-08-11 | 京东数字科技控股有限公司 | Log query method and device, electronic equipment and storage medium |
| CN111625423A (en) * | 2020-05-27 | 2020-09-04 | 网神信息技术（北京）股份有限公司 | Log processing method, apparatus, system, medium, and program |
| CN111639016A (en) * | 2020-05-29 | 2020-09-08 | 北京合力思腾科技股份有限公司 | Big data log analysis method and device and computer storage medium |
| CN111694793A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Log storage method and device and log query method and device |
| CN112084193A (en) * | 2020-09-04 | 2020-12-15 | 山东英信计算机技术有限公司 | Log query method, log storage method and related equipment |
| CN112307191A (en) * | 2020-11-03 | 2021-02-02 | 平安普惠企业管理有限公司 | Multi-system interactive log query method, device, equipment and storage medium |
| CN112532424A (en) * | 2020-11-03 | 2021-03-19 | 武汉悦学帮网络技术有限公司 | Service system, server and information processing method |
| CN112732759A (en) * | 2020-12-31 | 2021-04-30 | 青岛海尔科技有限公司 | Data processing method and device, storage medium and electronic device |
| CN112765118A (en) * | 2021-04-08 | 2021-05-07 | 北京优特捷信息技术有限公司 | Log query method, device, equipment and storage medium |
| CN112882992A (en) * | 2019-11-29 | 2021-06-01 | 北京百度网讯科技有限公司 | Method and apparatus for displaying information |
| CN112948328A (en) * | 2021-01-28 | 2021-06-11 | 长沙市到家悠享网络科技有限公司 | Retrieval method, device, equipment and medium of log data |
| CN112948334A (en) * | 2021-03-31 | 2021-06-11 | 建信金融科技有限责任公司 | Log processing method and device |
| CN113138896A (en) * | 2021-04-25 | 2021-07-20 | 中国工商银行股份有限公司 | Application running condition monitoring method, device and equipment |
| CN113157659A (en) * | 2021-05-14 | 2021-07-23 | 中国建设银行股份有限公司 | Log processing method and device |
| CN113177023A (en) * | 2021-04-19 | 2021-07-27 | 杭州海康威视系统技术有限公司 | Log retrieval method and device and electronic equipment |
| CN113326004A (en) * | 2021-06-10 | 2021-08-31 | 深圳市移卡科技有限公司 | Efficient log centralization method and device in cloud computing environment |
| WO2021189954A1 (en) * | 2020-10-12 | 2021-09-30 | 平安科技（深圳）有限公司 | Log data processing method and apparatus, computer device, and storage medium |
| CN113609162A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Query method, device, server and storage medium for operation records |
| CN113765694A (en) * | 2021-03-24 | 2021-12-07 | 北京京东拓先科技有限公司 | Log information display method and device, electronic equipment and computer readable medium |
| CN113778780A (en) * | 2020-11-27 | 2021-12-10 | 北京京东尚科信息技术有限公司 | Application stability determination method and device, electronic equipment and storage medium |
| CN113792036A (en) * | 2021-01-15 | 2021-12-14 | 北京沃东天骏信息技术有限公司 | Service data processing method, device, medium and electronic equipment |
| CN114095346A (en) * | 2020-08-04 | 2022-02-25 | 深圳云里物里科技股份有限公司 | Log collection method and system |
| CN114301769A (en) * | 2021-12-29 | 2022-04-08 | 杭州迪普信息技术有限公司 | Method and system for processing original flow data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778188A (en) * | 2014-02-24 | 2015-07-15 | 贵州电网公司信息通信分公司 | Distributed device log collection method |
| CN105447099A (en) * | 2015-11-11 | 2016-03-30 | 中国建设银行股份有限公司 | Log structured information extraction method and apparatus |
| CN107404658A (en) * | 2016-05-19 | 2017-11-28 | 中兴通讯股份有限公司 | A kind of interactive Web TV system and user data real time acquiring method |
| CN107451034A (en) * | 2017-08-17 | 2017-12-08 | 浪潮软件股份有限公司 | A kind of big data cluster log management apparatus, method and system |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| CN108804497A (en) * | 2018-04-02 | 2018-11-13 | 北京国电通网络技术有限公司 | A kind of big data analysis method based on daily record |
-
2018
- 2018-12-12 CN CN201811520640.XA patent/CN109800223A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104778188A (en) * | 2014-02-24 | 2015-07-15 | 贵州电网公司信息通信分公司 | Distributed device log collection method |
| CN105447099A (en) * | 2015-11-11 | 2016-03-30 | 中国建设银行股份有限公司 | Log structured information extraction method and apparatus |
| CN107404658A (en) * | 2016-05-19 | 2017-11-28 | 中兴通讯股份有限公司 | A kind of interactive Web TV system and user data real time acquiring method |
| CN107451034A (en) * | 2017-08-17 | 2017-12-08 | 浪潮软件股份有限公司 | A kind of big data cluster log management apparatus, method and system |
| CN107577588A (en) * | 2017-09-26 | 2018-01-12 | 北京中安智达科技有限公司 | A kind of massive logs data intelligence operational system |
| CN108804497A (en) * | 2018-04-02 | 2018-11-13 | 北京国电通网络技术有限公司 | A kind of big data analysis method based on daily record |
Cited By (47)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110569214A (en) * | 2019-08-02 | 2019-12-13 | 杭州云纪网络科技有限公司 | Index construction method and device for log file and electronic equipment |
| CN110569214B (en) * | 2019-08-02 | 2023-07-28 | 杭州云纪网络科技有限公司 | Index construction method and device for log file and electronic equipment |
| CN110674159A (en) * | 2019-09-02 | 2020-01-10 | 平安科技（深圳）有限公司 | Service query method and related device |
| WO2021043034A1 (en) * | 2019-09-02 | 2021-03-11 | 平安科技（深圳）有限公司 | Service query method, and related device |
| CN110730109A (en) * | 2019-10-12 | 2020-01-24 | 北京百度网讯科技有限公司 | Method and apparatus for generating information |
| CN110716910A (en) * | 2019-10-14 | 2020-01-21 | 中国建设银行股份有限公司 | Log management method, device, equipment and storage medium |
| CN110716910B (en) * | 2019-10-14 | 2022-10-14 | 中国建设银行股份有限公司 | Log management method, device, equipment and storage medium |
| CN110727641B (en) * | 2019-10-21 | 2023-10-27 | 中国民航信息网络股份有限公司 | Log searching method and device |
| CN110727641A (en) * | 2019-10-21 | 2020-01-24 | 中国民航信息网络股份有限公司 | Log searching method and device |
| CN110889068A (en) * | 2019-10-23 | 2020-03-17 | 北京达佳互联信息技术有限公司 | Method and device for assisting in consulting log and electronic equipment |
| CN111046010A (en) * | 2019-11-13 | 2020-04-21 | 泰康保险集团股份有限公司 | Log storage method, device, system, electronic equipment and computer readable medium |
| CN111047427A (en) * | 2019-11-26 | 2020-04-21 | 深圳市卡牛科技有限公司 | Data reporting method, device, server and storage medium |
| CN112882992A (en) * | 2019-11-29 | 2021-06-01 | 北京百度网讯科技有限公司 | Method and apparatus for displaying information |
| CN111144086A (en) * | 2019-12-20 | 2020-05-12 | 锐捷网络股份有限公司 | Log formatting method and device, electronic equipment and storage medium |
| CN111176762A (en) * | 2019-12-31 | 2020-05-19 | 北京奇艺世纪科技有限公司 | Batch processing program execution method and device, electronic equipment and storable medium |
| CN111241137A (en) * | 2020-01-08 | 2020-06-05 | 北京字节跳动网络技术有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN111241137B (en) * | 2020-01-08 | 2023-08-22 | 北京字节跳动网络技术有限公司 | Data processing method, device, electronic equipment and storage medium |
| CN111324510A (en) * | 2020-02-21 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Log processing method and device and electronic equipment |
| CN111324510B (en) * | 2020-02-21 | 2024-04-02 | 腾讯科技（深圳）有限公司 | Log processing method and device and electronic equipment |
| CN111522714A (en) * | 2020-04-20 | 2020-08-11 | 京东数字科技控股有限公司 | Log query method and device, electronic equipment and storage medium |
| CN111625423A (en) * | 2020-05-27 | 2020-09-04 | 网神信息技术（北京）股份有限公司 | Log processing method, apparatus, system, medium, and program |
| CN111639016A (en) * | 2020-05-29 | 2020-09-08 | 北京合力思腾科技股份有限公司 | Big data log analysis method and device and computer storage medium |
| CN111694793A (en) * | 2020-06-12 | 2020-09-22 | 北京金山云网络技术有限公司 | Log storage method and device and log query method and device |
| CN114095346A (en) * | 2020-08-04 | 2022-02-25 | 深圳云里物里科技股份有限公司 | Log collection method and system |
| CN112084193A (en) * | 2020-09-04 | 2020-12-15 | 山东英信计算机技术有限公司 | Log query method, log storage method and related equipment |
| WO2021189954A1 (en) * | 2020-10-12 | 2021-09-30 | 平安科技（深圳）有限公司 | Log data processing method and apparatus, computer device, and storage medium |
| CN112532424A (en) * | 2020-11-03 | 2021-03-19 | 武汉悦学帮网络技术有限公司 | Service system, server and information processing method |
| CN112307191A (en) * | 2020-11-03 | 2021-02-02 | 平安普惠企业管理有限公司 | Multi-system interactive log query method, device, equipment and storage medium |
| CN113778780A (en) * | 2020-11-27 | 2021-12-10 | 北京京东尚科信息技术有限公司 | Application stability determination method and device, electronic equipment and storage medium |
| CN113778780B (en) * | 2020-11-27 | 2024-05-17 | 北京京东尚科信息技术有限公司 | Application stability determining method and device, electronic equipment and storage medium |
| CN112732759B (en) * | 2020-12-31 | 2023-02-03 | 青岛海尔科技有限公司 | Data processing method and device, storage medium and electronic device |
| CN112732759A (en) * | 2020-12-31 | 2021-04-30 | 青岛海尔科技有限公司 | Data processing method and device, storage medium and electronic device |
| CN113792036A (en) * | 2021-01-15 | 2021-12-14 | 北京沃东天骏信息技术有限公司 | Service data processing method, device, medium and electronic equipment |
| CN112948328A (en) * | 2021-01-28 | 2021-06-11 | 长沙市到家悠享网络科技有限公司 | Retrieval method, device, equipment and medium of log data |
| CN113765694A (en) * | 2021-03-24 | 2021-12-07 | 北京京东拓先科技有限公司 | Log information display method and device, electronic equipment and computer readable medium |
| CN113765694B (en) * | 2021-03-24 | 2024-04-19 | 北京京东拓先科技有限公司 | Log information display method, device, electronic equipment and computer readable medium |
| CN112948334A (en) * | 2021-03-31 | 2021-06-11 | 建信金融科技有限责任公司 | Log processing method and device |
| CN112765118A (en) * | 2021-04-08 | 2021-05-07 | 北京优特捷信息技术有限公司 | Log query method, device, equipment and storage medium |
| CN113177023B (en) * | 2021-04-19 | 2023-07-25 | 杭州海康威视系统技术有限公司 | Log retrieval method and device and electronic equipment |
| CN113177023A (en) * | 2021-04-19 | 2021-07-27 | 杭州海康威视系统技术有限公司 | Log retrieval method and device and electronic equipment |
| CN113138896A (en) * | 2021-04-25 | 2021-07-20 | 中国工商银行股份有限公司 | Application running condition monitoring method, device and equipment |
| CN113157659A (en) * | 2021-05-14 | 2021-07-23 | 中国建设银行股份有限公司 | Log processing method and device |
| CN113326004B (en) * | 2021-06-10 | 2023-03-03 | 深圳市移卡科技有限公司 | Efficient log centralization method and device in cloud computing environment |
| CN113326004A (en) * | 2021-06-10 | 2021-08-31 | 深圳市移卡科技有限公司 | Efficient log centralization method and device in cloud computing environment |
| CN113609162A (en) * | 2021-07-14 | 2021-11-05 | 远景智能国际私人投资有限公司 | Query method, device, server and storage medium for operation records |
| CN113609162B (en) * | 2021-07-14 | 2023-09-26 | 远景智能国际私人投资有限公司 | Query method and device for operation records, server and storage medium |
| CN114301769A (en) * | 2021-12-29 | 2022-04-08 | 杭州迪普信息技术有限公司 | Method and system for processing original flow data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109800223A (en) | Log processing method, device, electronic equipment and storage medium | |
| US20170109657A1 (en) | Machine Learning-Based Model for Identifying Executions of a Business Process | |
| CN108182215B (en) | Structured Query Language (SQL) performance statistics method and device | |
| US20170109676A1 (en) | Generation of Candidate Sequences Using Links Between Nonconsecutively Performed Steps of a Business Process | |
| CN111294217B (en) | Alarm analysis method, device, system and storage medium | |
| CN109388637A (en) | Data warehouse information processing method, device, system, medium | |
| CN108572963A (en) | Information acquisition method and device | |
| US20180046956A1 (en) | Warning About Steps That Lead to an Unsuccessful Execution of a Business Process | |
| CN109359026A (en) | Log reporting method, device, electronic equipment and computer readable storage medium | |
| US20170109639A1 (en) | General Model for Linking Between Nonconsecutively Performed Steps in Business Processes | |
| CN110019213A (en) | Data managing method, device, electronic equipment and storage medium | |
| CN102929759A (en) | Business action monitoring operation time program | |
| CN110163457A (en) | A kind of abnormal localization method and device of operational indicator | |
| CN112686717B (en) | Data processing method and system for advertisement recall | |
| CN108074033A (en) | Processing method, system, electronic equipment and the storage medium of achievement data | |
| CN109547261A (en) | Service line switching method, device, electronic equipment and storage medium | |
| CN104657437A (en) | Monitoring method and monitoring device for promotion status data | |
| CN109597810A (en) | A kind of task cutting method, device, medium and electronic equipment | |
| CN110225076A (en) | File interaction method, apparatus, electronic equipment and storage medium | |
| CN109118225A (en) | Collectibles management method, device, medium and electronic equipment based on block chain | |
| CN108540439A (en) | Data analysis method and system, equipment and storage medium | |
| US11403313B2 (en) | Dynamic visualization of application and infrastructure components with layers | |
| US20170109637A1 (en) | Crowd-Based Model for Identifying Nonconsecutive Executions of a Business Process | |
| CN113836187A (en) | Data processing method, device, server and computer readable storage medium | |
| CN108337100A (en) | A kind of method and apparatus of cloud platform monitoring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |