CN109933505A - Log processing method, device, computer equipment and storage medium - Google Patents
Log processing method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN109933505A CN109933505A CN201910193751.2A CN201910193751A CN109933505A CN 109933505 A CN109933505 A CN 109933505A CN 201910193751 A CN201910193751 A CN 201910193751A CN 109933505 A CN109933505 A CN 109933505A
- Authority
- CN
- China
- Prior art keywords
- log
- business diary
- business
- data
- diary data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 87
- 230000000712 assembly Effects 0.000 claims abstract description 36
- 238000000429 assembly Methods 0.000 claims abstract description 36
- 238000000034 method Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 24
- 230000005540 biological transmission Effects 0.000 claims description 12
- 238000006073 displacement reaction Methods 0.000 claims description 12
- 230000008859 change Effects 0.000 claims description 8
- 238000004458 analytical method Methods 0.000 description 15
- 238000010586 diagram Methods 0.000 description 4
- 239000012634 fragment Substances 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 239000000047 product Substances 0.000 description 4
- 230000001419 dependent effect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000012800 visualization Methods 0.000 description 3
- 241000282372 Panthera onca Species 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000000737 periodic effect Effects 0.000 description 2
- 238000006116 polymerization reaction Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 238000012550 audit Methods 0.000 description 1
- 239000013065 commercial product Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Abstract
This application involves a kind of log processing method, device, computer equipment and storage mediums.Method includes: that log collector obtains business diary data, and the business diary data are transmitted to message system；The business diary data are transmitted to log processing pipeline by message queue by the message system；The log processing pipeline is handled the business diary data according to business demand, and by treated, business diary data are sent to distributed type assemblies；Business diary data that treated described in the distributed type assemblies storage simultaneously carry out log index.This log processing method not only improves only the utilization rate of log, and the mode of the log collection processing automated also significantly reduces the processing cost of log.
Description
Technical field
This application involves field of computer technology, more particularly to a kind of log processing method, device, computer equipment and
Storage medium.
Background technique
With the development of computer technology, operation system is also rapidly developed, the daily record data of each large enterprises
It is more and more.In the conventional technology, need to log in machine when checking log, and when there are more machines for a business module
When device, by checking that log orientation problem efficiency can be very low, and the complicated polymerization analysis of daily record data is highly dependent on fortune
The manual statistics for tieing up technical staff, needs powerful regular expression grounding in basic skills, such as: awk, sed (a kind of text flow editing machine)
Deng.
However, the more original mode such as this manual analyzing statistics for depending only on O&M, counts the day of hundred million ranks
Will item number consumes manpower and time cost is very high, and the mode dependent on O&M technical staff also can usually have log
Problem scattered, loss is low with log utilization rate.
Summary of the invention
Based on this, it is necessary in view of the above technical problems, provide a kind of log for capableing of log processing efficiency and utilization rate
Processing method, device, computer equipment and storage medium.
A kind of log processing method, which comprises
Log collector obtains business diary data, and the business diary data are transmitted to message system；
The business diary data are transmitted to log processing pipeline by message queue by the message system；
The log processing pipeline is handled the business diary data according to business demand, will treated business
Daily record data is sent to distributed type assemblies；
Business diary data that treated described in the distributed type assemblies storage simultaneously carry out log index.
A kind of log processing device, described device include:
Log acquisition module obtains business diary data for log collector, the business diary data is transmitted to
Message system；
The business diary data are transmitted to log by message queue for the message system by log transmission module
Handle pipeline；
Log processing module, for the log processing pipeline according to business demand to the business diary data at
Reason, by treated, business diary data are sent to distributed type assemblies；Business that treated described in the distributed type assemblies storage
Daily record data simultaneously carries out log index.
A kind of computer equipment, including memory, processor, the memory are stored with computer program, the processing
Device performs the steps of when executing the computer program
Log collector obtains business diary data, and the business diary data are transmitted to message system；
The business diary data are transmitted to log processing pipeline by message queue by the message system；
The log processing pipeline is handled the business diary data according to business demand, will treated business
Daily record data is sent to distributed type assemblies；
Business diary data that treated described in the distributed type assemblies storage simultaneously carry out log index.
A kind of computer readable storage medium, is stored thereon with computer program, and the computer program is held by processor
It is performed the steps of when row
Log collector obtains business diary data, and the business diary data are transmitted to message system；
The business diary data are transmitted to log processing pipeline by message queue by the message system；
The log processing pipeline is handled the business diary data according to business demand, will treated business
Daily record data is sent to distributed type assemblies；
Business diary data that treated described in the distributed type assemblies storage simultaneously carry out log index.
Above-mentioned log processing method, device, computer equipment and storage medium obtain business diary by log collector
Business diary data are transmitted to message system by data, and business diary data are transmitted the Summer Solstice or the Winter Solstice by message queue by message system
Will handles pipeline, and log processing pipeline handles business diary data according to business demand, will treated business diary
Data are sent to distributed type assemblies, and distributed type assemblies storage treated business diary data simultaneously carry out log index, work as needs
When using daily record data, then quick obtaining can be indexed by log to the log for wanting to obtain, to be directed to the daily record data
It the operation such as is analyzed or is shown, not only improve only the utilization rate of log, and the mode of the log collection processing automated
Significantly reduce the processing cost of log.
Detailed description of the invention
Fig. 1 is the applied environment figure of log processing method in one embodiment；
Fig. 2 is the flow diagram of log processing method in one embodiment；
Fig. 3 is the flow diagram of log processing method in another embodiment；
Fig. 4 is the structural block diagram of log processing device in one embodiment；
Fig. 5 is the internal structure chart of computer equipment in one embodiment.
Specific embodiment
It is with reference to the accompanying drawings and embodiments, right in order to which the objects, technical solutions and advantages of the application are more clearly understood
The application is further elaborated.It should be appreciated that specific embodiment described herein is only used to explain the application, not
For limiting the application.
Log processing method provided by the present application can be applied in application environment as shown in Figure 1.Wherein, log is adopted
Storage 102 is communicated with message system 104 by network, and log collector 102 is mounted on service server, will acquire
Business diary data be transmitted to message system 104, business diary data are then transmitted to log processing pipeline by message system 104
106, distributed type assemblies 108, final service daily record data are transmitted to after being handled by log processing pipeline 106 daily record data
Distributed type assemblies 108 are stored in processed log.Wherein, log collector 102 can be a filebeat (lightweight
For collecting and forwarding the collector of daily record data), service server can be, but not limited to be various personal computers, notes
This computer, smart phone, tablet computer and portable wearable device, it is (distributed that message system can be ckafka with 104
Message system), log processing pipeline 106 can be logstash, and distributed type assemblies 108 can be es cluster
(abbreviation of elasticsearch, the real-time search of distributed expandable and analysis engine).
In one embodiment, as shown in Fig. 2, providing a kind of log processing method, it is applied in Fig. 1 in this way
It is illustrated for terminal, comprising the following steps:
Step 202, log collector obtains business diary data, and business diary data are transmitted to message system.
The tool that log collector refers to dedicated for being acquired to business diary, log collector can be
Filebeat, the collector for being used to collect and forward daily record data of a lightweight.When log collector gets business day
Business diary data forwarding can be transmitted to message system after will data.Message system refers to by way of message queue pair
The system that data are transmitted, message system can be ckafka, a kind of distributed message system, for providing message queue
Service.Business diary data refer to the related daily record data with each business saved on service server.
In one embodiment, before log collector obtains business diary data, further includes: pacify log collector
Mounted in each service server, preassigned business diary file is monitored；When preassigned business diary file
Journal file displacement when change has occurred, obtain the business diary data of service server.
When needing capturing service daily record data, log collector can be mounted on each service server, that is, existed
Log collector is equipped on each service server.Service server refers to be connected by operation layer, and is stored with each business
The server of the corresponding operation log recording of operation.When acquiring log, some business diary files can be preassigned, i.e., in advance
It is fixed to determine the business diary file for needing to be monitored.Changed when monitoring these predesignated business diary file displacements
When change, then these business diary data changed are obtained.Business diary file displacement refers to the position of business diary content
It sets.For example 1 script of log is aaaa, i.e., displacement is 4, and after becoming aaaabb, i.e., displacement becomes 6, in this case, then can be recognized
Displacement for log 1 is changed, and log collector can get the daily record data of log 1.
In one embodiment, after business diary data are transmitted to message system, further includes: message system is to industry
Business daily record data is temporarily stored, when storage a length of default storage duration；Message system is to the industry beyond default storage duration
Business daily record data is deleted.
After the business diary data that log collector will acquire collocation are transmitted to message system, message system can be to business
Daily record data is temporarily stored.The configuration of temporary storage time can be carried out to message system, for example to be configurable to 2 small
When, i.e., message system can retain nearest log in 2 hours, and be more than that the default business diary data for storing duration will be deleted
It removes.Specifically, message system, when deleting expired daily record data, the deleting mechanism of use is the fragment batch according to message system
What amount was deleted, and be not to take deleting mechanism immediately.For example, the size of fragment is 1G, when fragment would not be to day less than 1G
Will carries out delete operation.That is, it is assumed that a length of 120 minutes when the default storage of setting, the size of data of fragment is at 120 points
1G can not be increased in clock, then daily record data will not be deleted immediately after reaching 120 minutes.
Step 204, business diary data are transmitted to log processing pipeline by message queue by message system.
Step 206, log processing pipeline is handled business diary data according to business demand, will treated business
Daily record data is sent to distributed type assemblies.
Step 208, distributed type assemblies storage treated business diary data and log index is carried out.
After message system receives the business diary data of log collector transmission, message system can pass through message team
Business diary data are transmitted to log processing pipeline by the mode of column.Log processing pipeline can be logstash, logstash
Be a kind of log processing pipeline, data can be acquired from multiple sources simultaneously, parse data and change data, so as to more rapidly and
More easily analyze data.Log processing pipeline can be handled daily record data according to business demand.
After log processing pipeline is handled to obtain treated business diary data to daily record data, log processing pipe
Road can business diary data be sent to distributed type assemblies by treated, and distributed type assemblies are responsible for the log processing pipe of upstream
The daily record data that road transmits does log index, and distributed type assemblies are also the place that business diary data finally store.Distribution
Formula cluster, which carries out log index to business diary data, to be advantageous in that, business diary data can be carried out to orderly storage.
Distributed type assemblies are clustered deploy(ment)s, and the service integration carried out towards enterprise provides full-text search, log analysis, data analysis etc.
Function.
In one embodiment, above-mentioned log processing method further include: log number to be presented is obtained by log index
According to；Daily record data is shown by display platform.
It, can be by the log of distributed type assemblies point after carrying out log index to business diary data by distributed type assemblies
Analysis result is shown by visualizing platform, and does post analysis.Display platform can be a kibana (analysis
And Visualization Platform).External interface can also be provided according to the business of company, pass through the available number arrived of this external interface
It is many according to having, for example PV (page browsing amount), UV (user's amount of access), the API of the daily request amount of coreuser, Jaguar system ask
Seeking time and http status code statistics etc..The visualization that a real time monitoring can also be made according to these analysis data is flat
Platform is also based on the operation data of service conditions periodic statistical output, or according to the external api interface of offer, realizes product
With all kinds of complicated aggregate query demands of the daily record data of operation personnel, a large amount of daily record datas are made the best use of everything.
Above-mentioned log processing method obtains business diary data by log collector, business diary data is transmitted to
Business diary data are transmitted to log processing pipeline, log processing pipeline root by message queue by message system, message system
Business diary data are handled according to business demand, by treated, business diary data are sent to distributed type assemblies, distribution
Formula cluster-based storage treated business diary data simultaneously carry out log index, when needing to use daily record data, then can pass through
Log indexes quick obtaining to the log for wanting to obtain, to be analyzed or be shown etc. for the daily record data and operate, not only
The utilization rate of log is improved only, and the mode of the log collection processing automated also significantly reduces being processed into for log
This.
In one embodiment, log processing pipeline is handled business diary data according to business demand, comprising: day
Will handles pipeline and business diary data is parsed into predetermined format；The business diary data after predetermined format will be parsed into according to industry
Business demand carries out field processing, obtains the statistical data of each field.
Specifically, log processing pipeline is when handling business diary data according to business demand, log processing pipe
Business diary data first can be parsed into predetermined format by road, and predetermined format refers to the format defined in advance by technical staff, such as
Predetermined format can be Jason format.After the parsing for carrying out format, can according to business demand to business diary data into
The processing of row field.For example business demand is to count the frequency of occurrences of each field, then log processing pipeline can be to business diary
The field of every a line log in data is split, and counts the frequency of occurrence of each field, or each field pair of statistics
The corresponding statistical data of each field can be obtained in the number that the value answered occurs in this way.
In one embodiment, above-mentioned log processing method further include: message system obtains the slow log in database, will
Slow log transmission is to log processing pipeline；Log processing pipeline according to business demand to slow log and business diary data at
Reason.
Slow log refers to that the response time is more than the sql sentence of threshold values in MySQL.For example, response lag is 10s, then transporting
The row time is more than that the sql sentence of 10s is logged into slow log, and some characteristic values of this subsidiary log, such as sql sentence
It holds, executes user, executes time and log generation time etc..It, can be by writing when slow log is directed into message system
Python script is imported into ckafka in by the slow log of mysql database, that is, is directed into message system, so as into
Enter log processing process.After slow log is imported into message system, log processing pipeline can be by slow log and other business days
Will equally carries out log processing step.
The collection of log processing flow chart as shown in Figure 3, log needs the installation log collector on service server
Filebeat, it can monitor the journal file for being designated to collect, and record log file displacement (offset) moment, when log text
When part displacement changes, filebeat can be collected daily record data, and pass through designated port and message system
Ckafka establishes connection, and log transmission is done to the processing of next step to message system.Message system is receiving business diary number
According to rear, daily record data can be cached.On the one hand, due to business diary substantial amounts, es distributed type assemblies pressure is concurrently squeezed into
Power can be very big；On the other hand, when es distributed type assemblies enter maintenance state, log can lose.So needing day
Will is first put into message system ckafka and caches, and downstream log processing pipeline logstash consumes this according still further to message queue
A little logs.
Log processing pipeline logstash reads log from message system ckafka, and to log field carry out processing and
Processing, and by the distributed es cluster in treated log is sent to downstream.Logstash is clustered deploy(ment), multiple can be consumed
End carries out log cutting process together.Logstash is responsible for acquiring and parsing log, and log is parsed into the format of needs
It is stored in es cluster.Logstash provides many very powerful plug-in units, these plug-in units can be effectively log information
It is converted into the format needed.After log is parsed, can more intuitively monitor control index, and index value carries out statistics and analysis, more
Quickly and more easily analyze data.
The log analysis result of es cluster can be come out by visualizing by kibana Visualization Platform, and
The post analysis for doing data provides external interface based on network service is treasured, effectively improves log utilization efficiency.For example,
The daily request amount of coreuser, the PV (page browsing amount) of Jaguar system, UV (user's amount of access), API request time and http
Status code statistics etc..Daily paper can also be runed, based on the operation data of service conditions periodic statistical output, can also be provided outer
Portion API realizes all kinds of complicated aggregate query demands of the daily record data of product and operation personnel, a large amount of daily record data object is use up it
With.
This centralized log platform voluntarily established, is greatly improved O&M, exploitation and data department to log
Inquiry and analysis efficiency, tracked including: fault location, problem discs and effect, to comprehensively improve log
Utilization rate.And product and operation are no longer highly dependent on O&M to the complicated polymerization analysis of user behavior, can also pass through log
Platform realizes service-oriented key performance tracking.For example, provide data source outgoing traffic operation daily paper, real-time monitoring platform,
New product user behavior analysis and interim log analysis statement of requirements, depth excavate the value of log big data.Self-built day
Will manages the cost of platform compared with mainstream log commercial product, and expense is greatly reduced, and has saved log management cost.Separately
Outside, this log processing method can also aid in technical staff and system effectively guarded, safeguard, optimize and improved, than
As externally provided api interface, realizes the demand that each post personnel analyzes daily record data, greatly reduce other systems to data
The acquisition cost of index, while can also aid in safety and closing the work such as rule audit, it escorts for the information security of enterprise.
It should be understood that although each step in the flow chart of Fig. 2-3 is successively shown according to the instruction of arrow,
These steps are not that the inevitable sequence according to arrow instruction successively executes.Unless expressly stating otherwise herein, these steps
Execution there is no stringent sequences to limit, these steps can execute in other order.Moreover, at least one in Fig. 2-3
Part steps may include that perhaps these sub-steps of multiple stages or stage are not necessarily in synchronization to multiple sub-steps
Completion is executed, but can be executed at different times, the execution sequence in these sub-steps or stage is also not necessarily successively
It carries out, but can be at least part of the sub-step or stage of other steps or other steps in turn or alternately
It executes.
In one embodiment, as shown in figure 4, providing a kind of log processing device, comprising:
Log acquisition module 402 obtains business diary data for log collector, business diary data is transmitted to and are disappeared
Breath system.
Business diary data are transmitted to log processing by message queue for message system by log transmission module 404
Pipeline.
Log processing module 406 is handled business diary data according to business demand for log processing pipeline, will
Treated, and business diary data are sent to distributed type assemblies；Distributed type assemblies storage treated business diary data simultaneously carry out
Log index.
In one embodiment, above-mentioned log acquisition module 402 is also used to for log collector being mounted on each business clothes
Business device, is monitored preassigned business diary file；When the journal file of preassigned business diary file is displaced
When change has occurred, the business diary data of service server are obtained.
In one embodiment, it is temporary to the progress of business diary data to be also used to message system for above-mentioned log transmission module 404
When store, when storage a length of default storage duration；Message system deletes the business diary data beyond default storage duration
It removes.
In one embodiment, above-mentioned log processing module 406 is also used to log processing pipeline for business diary data solution
Analyse into predetermined format；The business diary data after predetermined format will be parsed into, field processing is carried out according to business demand, obtained every
The statistical data of a field.
In one embodiment, above-mentioned log processing device further includes slow log transmission module (not shown), is used for
Message system obtains the slow log in database, by slow log transmission to log processing pipeline；Log processing pipeline is according to business
Demand handles slow log and business diary data.
In one embodiment, above-mentioned log processing device further includes data display module (not shown), for leading to
It crosses log index and obtains daily record data to be presented；Daily record data is shown by display platform.
Specific about log processing device limits the restriction that may refer to above for log processing method, herein not
It repeats again.Modules in above-mentioned log processing device can be realized fully or partially through software, hardware and combinations thereof.On
Stating each module can be embedded in the form of hardware or independently of in the processor in computer equipment, can also store in a software form
In memory in computer equipment, the corresponding operation of the above modules is executed in order to which processor calls.
In one embodiment, a kind of computer equipment is provided, which can be server, internal junction
Composition can be as shown in Figure 5.The computer equipment include by system bus connect processor, memory, network interface and
Database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The memory packet of the computer equipment
Include non-volatile memory medium, built-in storage.The non-volatile memory medium is stored with operating system, computer program and data
Library.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.The calculating
The database of machine equipment is for storing the relevant data of log.The network interface of the computer equipment is used for logical with external terminal
Cross network connection communication.To realize a kind of log processing method when the computer program is executed by processor.
It will be understood by those skilled in the art that structure shown in Fig. 5, only part relevant to application scheme is tied
The block diagram of structure does not constitute the restriction for the computer equipment being applied thereon to application scheme, specific computer equipment
It may include perhaps combining certain components or with different component layouts than more or fewer components as shown in the figure.
In one embodiment, a kind of computer equipment, including memory and processor are provided, is stored in memory
Computer program, the processor perform the steps of log collector and obtain business diary data when executing computer program, will
Business diary data are transmitted to message system；Business diary data are transmitted to log processing pipe by message queue by message system
Road；Log processing pipeline is handled business diary data according to business demand, and by treated, business diary data are sent
To distributed type assemblies；Distributed type assemblies storage treated business diary data simultaneously carry out log index.
In one embodiment, before log collector obtains business diary data, processor executes computer program
When also perform the steps of log collector be mounted on each service server, to preassigned business diary file into
Row monitoring；When change has occurred in the displacement of the journal file of preassigned business diary file, the industry of service server is obtained
Business daily record data.
In one embodiment, after business diary data are transmitted to message system, processor executes computer journey
Message system is also performed the steps of when sequence temporarily to store business diary data, when storage a length of default storage duration；
Message system deletes the business diary data beyond default storage duration.
In one embodiment, log processing pipeline is handled business diary data according to business demand, comprising: day
Will handles pipeline and business diary data is parsed into predetermined format；The business diary data after predetermined format will be parsed into according to industry
Business demand carries out field processing, obtains the statistical data of each field.
In one embodiment, message system is also performed the steps of when processor executes computer program obtains data
Slow log in library, by slow log transmission to log processing pipeline；Log processing pipeline is according to business demand to slow log and industry
Business daily record data is handled.
In one embodiment, it also performs the steps of to index by log when processor executes computer program and obtain
Daily record data to be presented；Daily record data is shown by display platform.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
Machine program performs the steps of log collector and obtains business diary data when being executed by processor, business diary data are passed
Transport to message system；Business diary data are transmitted to log processing pipeline by message queue by message system；Log processing pipe
Road is handled business diary data according to business demand, and by treated, business diary data are sent to distributed type assemblies；
Distributed type assemblies storage treated business diary data simultaneously carry out log index.
In one embodiment, before log collector obtains business diary data, computer program is held by processor
It is also performed the steps of when row and log collector is mounted on each service server, to preassigned business diary file
It is monitored；When change has occurred in the displacement of the journal file of preassigned business diary file, service server is obtained
Business diary data.
In one embodiment, after business diary data are transmitted to message system, computer program is by processor
It also performs the steps of message system when execution temporarily to store business diary data, when storage when a length of default storage
It is long；Message system deletes the business diary data beyond default storage duration.
In one embodiment, log processing pipeline is handled business diary data according to business demand, comprising: day
Will handles pipeline and business diary data is parsed into predetermined format；The business diary data after predetermined format will be parsed into according to industry
Business demand carries out field processing, obtains the statistical data of each field.
In one embodiment, message system is also performed the steps of when computer program is executed by processor obtains number
According to the slow log in library, by slow log transmission to log processing pipeline；Log processing pipeline according to business demand to slow log and
Business diary data are handled.
In one embodiment, it also performs the steps of when computer program is executed by processor and is obtained by log index
Take daily record data to be presented；Daily record data is shown by display platform.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Instruct relevant hardware to complete by computer program, computer program to can be stored in a non-volatile computer readable
It takes in storage medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, this Shen
Please provided by any reference used in each embodiment to memory, storage, database or other media, may each comprise
Non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
Each technical characteristic of above embodiments can be combined arbitrarily, for simplicity of description, not to above-described embodiment
In each technical characteristic it is all possible combination be all described, as long as however, the combination of these technical characteristics be not present lance
Shield all should be considered as described in this specification.
Above embodiments only express the several embodiments of the application, and the description thereof is more specific and detailed, but can not
Therefore it is construed as limiting the scope of the patent.It should be pointed out that for those of ordinary skill in the art,
Under the premise of not departing from the application design, various modifications and improvements can be made, these belong to the protection scope of the application.
Therefore, the scope of protection shall be subject to the appended claims for the application patent.
Claims (10)
1. a kind of log processing method, which comprises
Log collector obtains business diary data, and the business diary data are transmitted to message system；
The business diary data are transmitted to log processing pipeline by message queue by the message system；
The log processing pipeline is handled the business diary data according to business demand, will treated business diary
Data are sent to distributed type assemblies；
Business diary data that treated described in the distributed type assemblies storage simultaneously carry out log index.
2. the method according to claim 1, wherein the log collector obtain business diary data it
Before, further includes:
Log collector is mounted on each service server, preassigned business diary file is monitored；
When change has occurred in the displacement of the journal file of the preassigned business diary file, the service server is obtained
Business diary data.
3. the method according to claim 1, wherein by the business diary data be transmitted to message system it
Afterwards, further includes:
The message system temporarily stores the business diary data, when storage a length of default storage duration；
The message system deletes the business diary data beyond the default storage duration.
4. the method according to claim 1, wherein the log processing pipeline is according to business demand to the industry
Business daily record data is handled, comprising:
The business diary data are parsed into predetermined format by the log processing pipeline；
The business diary data after predetermined format will be parsed into, field processing is carried out according to business demand, obtains the system of each field
It counts.
5. the method according to claim 1, wherein the method also includes:
The message system obtains the slow log in database, by the slow log transmission to log processing pipeline；
The log processing pipeline is handled the slow log and the business diary data according to business demand.
6. the method according to claim 1, wherein the method also includes:
Daily record data to be presented is obtained by log index；
The daily record data is shown by display platform.
7. a kind of log processing device, which is characterized in that described device includes:
Log acquisition module obtains business diary data for log collector, the business diary data is transmitted to message
System；
The business diary data are transmitted to log processing by message queue for the message system by log transmission module
Pipeline；
Log processing module is handled the business diary data according to business demand for the log processing pipeline,
By treated, business diary data are sent to distributed type assemblies；Business diary that treated described in the distributed type assemblies storage
Data simultaneously carry out log index.
8. device according to claim 7, which is characterized in that the log acquisition module is also used to pacify log collector
Mounted in each service server, preassigned business diary file is monitored；When the preassigned business diary
When change has occurred in the journal file displacement of file, the business diary data of the service server are obtained.
9. a kind of computer equipment, including memory and processor, the memory are stored with computer program, feature exists
In the step of processor realizes any one of claims 1 to 6 the method when executing the computer program.
10. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
The step of method described in any one of claims 1 to 6 is realized when being executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910193751.2A CN109933505A (en) | 2019-03-14 | 2019-03-14 | Log processing method, device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910193751.2A CN109933505A (en) | 2019-03-14 | 2019-03-14 | Log processing method, device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109933505A true CN109933505A (en) | 2019-06-25 |
Family
ID=66987247
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910193751.2A Pending CN109933505A (en) | 2019-03-14 | 2019-03-14 | Log processing method, device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109933505A (en) |
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110399357A (en) * | 2019-06-28 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus and system for managing the log of big data component concentratedly |
| CN110515898A (en) * | 2019-07-31 | 2019-11-29 | 济南浪潮数据技术有限公司 | A kind of log processing method and device |
| CN110688354A (en) * | 2019-09-30 | 2020-01-14 | 深圳证券交易所 | Analysis method of slow log file in database, terminal and storage medium |
| CN111143286A (en) * | 2019-12-13 | 2020-05-12 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
| CN111258979A (en) * | 2020-01-16 | 2020-06-09 | 山东大学 | Cloud protection log system and working method thereof |
| CN111352903A (en) * | 2020-03-13 | 2020-06-30 | 京东方科技集团股份有限公司 | Log management platform, log management method, medium, and electronic device |
| CN111897790A (en) * | 2020-07-31 | 2020-11-06 | 平安普惠企业管理有限公司 | Wind control log collection method and device, electronic equipment and storage medium |
| CN111949705A (en) * | 2020-07-31 | 2020-11-17 | 上海中通吉网络技术有限公司 | Slow query statistical analysis method, device and system |
| CN112000617A (en) * | 2020-08-07 | 2020-11-27 | 北京浪潮数据技术有限公司 | Log preprocessing method, device, equipment and readable storage medium |
| CN112395157A (en) * | 2020-11-13 | 2021-02-23 | 广州至真信息科技有限公司 | Audit log obtaining method and device, computer equipment and storage medium |
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN112948334A (en) * | 2021-03-31 | 2021-06-11 | 建信金融科技有限责任公司 | Log processing method and device |
| CN113010399A (en) * | 2021-04-16 | 2021-06-22 | 携程旅游网络技术（上海）有限公司 | Log data processing method, system, device and medium |
| CN113326237A (en) * | 2021-06-16 | 2021-08-31 | 深圳市明源云科技有限公司 | Log data processing method and device, terminal device and storage medium |
| CN113342564A (en) * | 2021-06-25 | 2021-09-03 | 阿波罗智联(北京)科技有限公司 | Log auditing method and device, electronic equipment and medium |
| CN114238258A (en) * | 2021-11-30 | 2022-03-25 | 企查查科技有限公司 | Database data processing method and device, computer equipment and storage medium |
| CN114598597A (en) * | 2022-02-24 | 2022-06-07 | 烽台科技（北京）有限公司 | Multi-source log analysis method and device, computer equipment and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
| CN105243147A (en) * | 2015-10-22 | 2016-01-13 | 浪潮(北京)电子信息产业有限公司 | Slow query log management method and system of MySQL database |
| CN106452819A (en) * | 2015-08-13 | 2017-02-22 | 腾讯科技（深圳）有限公司 | Data acquisition system and data acquisition method |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107273267A (en) * | 2017-06-09 | 2017-10-20 | 环球智达科技（北京）有限公司 | Log analysis method based on elastic components |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
-
2019
- 2019-03-14 CN CN201910193751.2A patent/CN109933505A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103838867A (en) * | 2014-03-20 | 2014-06-04 | 网宿科技股份有限公司 | Log processing method and device |
| CN106452819A (en) * | 2015-08-13 | 2017-02-22 | 腾讯科技（深圳）有限公司 | Data acquisition system and data acquisition method |
| CN105243147A (en) * | 2015-10-22 | 2016-01-13 | 浪潮(北京)电子信息产业有限公司 | Slow query log management method and system of MySQL database |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| CN107273267A (en) * | 2017-06-09 | 2017-10-20 | 环球智达科技（北京）有限公司 | Log analysis method based on elastic components |
| CN107180116A (en) * | 2017-06-28 | 2017-09-19 | 努比亚技术有限公司 | A kind of data synchronizing processing method, mobile terminal and computer-readable recording medium |
| CN107622084A (en) * | 2017-08-10 | 2018-01-23 | 深圳前海微众银行股份有限公司 | Blog management method, system and computer-readable recording medium |
Cited By (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110399357A (en) * | 2019-06-28 | 2019-11-01 | 苏州浪潮智能科技有限公司 | A kind of method, apparatus and system for managing the log of big data component concentratedly |
| CN110515898A (en) * | 2019-07-31 | 2019-11-29 | 济南浪潮数据技术有限公司 | A kind of log processing method and device |
| CN110515898B (en) * | 2019-07-31 | 2022-04-22 | 济南浪潮数据技术有限公司 | Log processing method and device |
| CN110688354A (en) * | 2019-09-30 | 2020-01-14 | 深圳证券交易所 | Analysis method of slow log file in database, terminal and storage medium |
| CN110688354B (en) * | 2019-09-30 | 2022-05-03 | 深圳证券交易所 | Analysis method of slow log file in database, terminal and storage medium |
| CN112866319B (en) * | 2019-11-28 | 2023-10-13 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN111143286A (en) * | 2019-12-13 | 2020-05-12 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
| CN111143286B (en) * | 2019-12-13 | 2024-01-09 | 无锡华云数据技术服务有限公司 | Cloud platform log management method and system |
| CN111258979B (en) * | 2020-01-16 | 2022-04-15 | 山东大学 | Cloud protection log system and working method thereof |
| CN111258979A (en) * | 2020-01-16 | 2020-06-09 | 山东大学 | Cloud protection log system and working method thereof |
| CN111352903A (en) * | 2020-03-13 | 2020-06-30 | 京东方科技集团股份有限公司 | Log management platform, log management method, medium, and electronic device |
| CN111897790A (en) * | 2020-07-31 | 2020-11-06 | 平安普惠企业管理有限公司 | Wind control log collection method and device, electronic equipment and storage medium |
| CN111949705A (en) * | 2020-07-31 | 2020-11-17 | 上海中通吉网络技术有限公司 | Slow query statistical analysis method, device and system |
| CN112000617B (en) * | 2020-08-07 | 2022-08-12 | 北京浪潮数据技术有限公司 | Log preprocessing method, device, equipment and readable storage medium |
| CN112000617A (en) * | 2020-08-07 | 2020-11-27 | 北京浪潮数据技术有限公司 | Log preprocessing method, device, equipment and readable storage medium |
| CN112395157B (en) * | 2020-11-13 | 2023-08-08 | 广州至真信息科技有限公司 | Audit log acquisition method and device, computer equipment and storage medium |
| CN112395157A (en) * | 2020-11-13 | 2021-02-23 | 广州至真信息科技有限公司 | Audit log obtaining method and device, computer equipment and storage medium |
| CN112948334A (en) * | 2021-03-31 | 2021-06-11 | 建信金融科技有限责任公司 | Log processing method and device |
| CN113010399A (en) * | 2021-04-16 | 2021-06-22 | 携程旅游网络技术（上海）有限公司 | Log data processing method, system, device and medium |
| CN113326237A (en) * | 2021-06-16 | 2021-08-31 | 深圳市明源云科技有限公司 | Log data processing method and device, terminal device and storage medium |
| CN113342564B (en) * | 2021-06-25 | 2023-12-12 | 阿波罗智联(北京)科技有限公司 | Log auditing method and device, electronic equipment and medium |
| CN113342564A (en) * | 2021-06-25 | 2021-09-03 | 阿波罗智联(北京)科技有限公司 | Log auditing method and device, electronic equipment and medium |
| CN114238258A (en) * | 2021-11-30 | 2022-03-25 | 企查查科技有限公司 | Database data processing method and device, computer equipment and storage medium |
| CN114238258B (en) * | 2021-11-30 | 2024-02-20 | 企查查科技股份有限公司 | Database data processing method, device, computer equipment and storage medium |
| CN114598597A (en) * | 2022-02-24 | 2022-06-07 | 烽台科技（北京）有限公司 | Multi-source log analysis method and device, computer equipment and medium |
| CN114598597B (en) * | 2022-02-24 | 2023-12-01 | 烽台科技（北京）有限公司 | Multisource log analysis method, multisource log analysis device, computer equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109933505A (en) | Log processing method, device, computer equipment and storage medium | |
| US11074560B2 (en) | Tracking processed machine data | |
| CN107861859B (en) | Log management method and system based on micro-service architecture | |
| CN103914485B (en) | System and method for remotely collecting, retrieving and displaying application system logs | |
| CN110909063B (en) | User behavior analysis method and device, application server and storage medium | |
| US8095690B2 (en) | Machine-readable medium for storing a stream data processing program and computer system | |
| CN108268485B (en) | Log real-time analysis method and system | |
| CN111125260A (en) | Data synchronization method and system based on SQL Server | |
| CN104182506A (en) | Log management method | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN105119762A (en) | System and method of cloud platform for realizing transaction playback and transaction reworking based on logs | |
| CN111245672A (en) | Monitoring method and system for general extensible tracking service full link | |
| CN110908883A (en) | User portrait data monitoring method, system, equipment and storage medium | |
| CN112948492A (en) | Data processing system, method and device, electronic equipment and storage medium | |
| CN112181931A (en) | Big data system link tracking method and electronic equipment | |
| CN113590556A (en) | Database-based log processing method, device and equipment | |
| CN106407429A (en) | File tracking method, device and system | |
| Murugesan et al. | Audit log management in MongoDB | |
| CN106919566A (en) | A kind of query statistic method and system based on mass data | |
| CN110309206B (en) | Order information acquisition method and system | |
| CN107577809A (en) | Offline small documents processing method and processing device | |
| CN111209314A (en) | System for processing massive log data of power information system in real time | |
| CN115840656A (en) | Automatic operation and maintenance method and system for application program based on fault self-healing | |
| CN112579406A (en) | Log call chain generation method and device | |
| CN112148562B (en) | Interface relation analysis method based on distributed system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190625 |