CN110134648A - Log processing method, device, equipment, system and computer readable storage medium - Google Patents
Log processing method, device, equipment, system and computer readable storage medium Download PDFInfo
- Publication number
- CN110134648A CN110134648A CN201910427734.0A CN201910427734A CN110134648A CN 110134648 A CN110134648 A CN 110134648A CN 201910427734 A CN201910427734 A CN 201910427734A CN 110134648 A CN110134648 A CN 110134648A
- Authority
- CN
- China
- Prior art keywords
- log
- processed
- daily record
- record data
- search server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/144—Query formulation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The embodiment of the present invention provides a kind of log processing method, device, equipment, system and computer readable storage medium, this method comprises: receiving daily record data to be processed；Distributed data base is written into the daily record data to be processed；Monitor whether the daily record data in the distributed data base updates, wherein described update includes increasing the daily record data to be processed newly；When daily record data in the distributed data base updates, the daily record data updated in the distributed data base is synchronized to search server, so that described search server searches the corresponding daily record data of the inquiry request after receiving inquiry request.The embodiment of the present invention can be by being synchronized to search server for daily record data, so that the write-in and inquiry to daily record data are separated from each other, to improve the response speed of the inquiry to daily record data.
Description
Technical field
The present embodiments relate to field of computer technology more particularly to a kind of log processing method, device, equipment, it is
System and computer readable storage medium.
Background technique
In electric business system, the retrieval and inquisition to system business log be it is important, it can reflect out from side
Logging in system by user frequency, number, checks the information such as product duration at access product preference, to provide accurate message push.
Traditional log processing method, mostly uses greatly front and back end mode, the page operation of front end receiver user, rear end according to
Operation information is directly added into database, and the mode of inquiry log is realized also by the form of direct inquiry database.
But as system business ability is increasing, amount of access constantly increases, and the data volume of business diary also can be increasingly
Greatly.The data volume increase of log will lead to the increase of database manipulation number, database disk read-write speed and network transmission
It can be busy.It is handled according to current log processing mode, the response time in terms of the retrieval and inquisition of log also can mutually strain
Long, response speed is slow.
Summary of the invention
The embodiment of the present invention provides a kind of log processing method, device, equipment, system and computer readable storage medium,
It is slow to the response speed of inquiry in a manner of current log processing to solve the problems, such as.
In a first aspect, the embodiment of the present invention provides a kind of log processing method, comprising:
Receive daily record data to be processed；
Distributed data base is written into the daily record data to be processed；
Monitor whether the daily record data in the distributed data base updates, wherein described update includes newly-increased institute
State daily record data to be processed；
When daily record data in the distributed data base updates, day for will being updated in the distributed data base
Will data are synchronized to search server, so that described search server searches the inquiry request pair after receiving inquiry request
The daily record data answered.
In a kind of possible embodiment, before reception daily record data to be processed, further includes:
Subscribe to the default log topic in Message Queuing system；
It is described to receive daily record data to be processed, comprising:
It monitors under the default log topic with the presence or absence of log information to be processed, wherein the log to be processed
Message includes the daily record data to be processed；
Under monitoring the default log topic there are when the log information to be processed, from the message queue system
The log information to be processed is obtained in system, and the log number to be processed is extracted from the log information to be processed
According to.
In a kind of possible embodiment, the log information to be processed is that log generating device generates and sends
The log information under the default log topic in the Message Queuing system.
It is described that data updated in the distributed data base are synchronized to search in a kind of possible embodiment
Server, comprising:
Data updated in the distributed data base are synchronized to described search service by data transport service DTS
Device.
Second aspect, the embodiment of the present invention provide a kind of log processing device, comprising:
Receiving module, for receiving daily record data to be processed；
Writing module, for distributed data base to be written in the daily record data to be processed；
Whether monitoring modular, the daily record data for monitoring in the distributed data base update, wherein it is described more
New includes increasing the daily record data to be processed newly；
Synchronization module, when being updated for the daily record data in the distributed data base, by the distributed number
It is synchronized to search server according to the daily record data updated in library, so that described search server is searched after receiving inquiry request
The corresponding daily record data of the inquiry request.
Further include subscribing module in a kind of possible embodiment, the subscribing module is used for:
Subscribe to the default log topic in Message Queuing system；
The receiving module is used for:
It monitors under the default log topic with the presence or absence of log information to be processed, wherein the log to be processed
Message includes the daily record data to be processed；
Under monitoring the default log topic there are when the log information to be processed, from the message queue system
The log information to be processed is obtained in system, and the log number to be processed is extracted from the log information to be processed
According to.
In a kind of possible embodiment, the log information to be processed is that log generating device generates and sends
The log information under the default log topic in the Message Queuing system.
In a kind of possible embodiment, the synchronization module is used for:
Data updated in the distributed data base are synchronized to described search service by data transport service DTS
Device.
The third aspect, the embodiment of the present invention provide a kind of log processing equipment, comprising: at least one processor and storage
Device；
The memory stores computer executed instructions；
At least one described processor executes the computer executed instructions of memory storage so that it is described at least one
Processor executes log processing method described in the various possible embodiments of first aspect and first aspect as above.
Fourth aspect, the embodiment of the present invention provide a kind of log processing system, comprising: the day as described in the third aspect as above
Will processing equipment and search server；
Described search server after receiving inquiry request for searching the corresponding daily record data of the inquiry request.
It further include log generating device and log query equipment in a kind of possible embodiment；
The log generating device is sent for generating log information to be processed, and by the log information to be processed
Under default log topic into the Message Queuing system, wherein the log information to be processed includes day to be processed
Will data；
The log query equipment is used to receive the inquiry instruction of user's input, is generated according to the inquiry instruction corresponding
Inquiry request, and described search server is sent by the inquiry request, receiving looking into for described search server return
After asking result, the query result is shown.
5th aspect, the embodiment of the present invention provide a kind of computer readable storage medium, the computer-readable storage medium
Computer executed instructions are stored in matter, when processor execute the computer executed instructions when, realize first aspect as above with
And log processing method described in the various possible embodiments of first aspect.
Log processing method, device, equipment, system and computer readable storage medium provided in this embodiment, by connecing
Receive daily record data to be processed；Distributed data base is written into daily record data to be processed；Monitor the day in distributed data base
Whether will data update, wherein updating includes increasing daily record data to be processed newly；Log number in distributed data base
When according to updating, the daily record data updated in distributed data base is synchronized to search server, so that search server exists
The corresponding daily record data of inquiry request is searched after receiving inquiry request, it can be by the way that daily record data be synchronized to search service
Device, so that the write-in and inquiry to daily record data are separated from each other, to improve the response speed of the inquiry to daily record data.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is this hair
Bright some embodiments for those of ordinary skill in the art without any creative labor, can be with
It obtains other drawings based on these drawings.
Fig. 1 is the configuration diagram for the log processing system that one embodiment of the invention provides；
Fig. 2 is the flow diagram for the log processing method that one embodiment of the invention provides；
Fig. 3 is the flow diagram for the log processing method that further embodiment of this invention provides；
Fig. 4 is the schematic diagram of a log processing process provided in an embodiment of the present invention；
Fig. 5 is the structural schematic diagram for the log processing device that one embodiment of the invention provides；
Fig. 6 is the structural schematic diagram for the log processing device that further embodiment of this invention provides；
Fig. 7 is the hardware structural diagram for the log processing equipment that one embodiment of the invention provides.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art
Every other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.
Fig. 1 is the configuration diagram for the log processing system that one embodiment of the invention provides.As shown in Figure 1, the present embodiment
The log processing system of offer includes that log generating device 11, log processing equipment 12, search server 13 and log query are set
Standby 14.It is illustrated by taking the log processing to electric business system as an example, but is not intended as limiting.Log generating device 11 can be electricity
Quotient system system, log query equipment can be electric business system and/log query system.Log processing equipment 12 receives log generation and sets
Standby 11 daily record datas sent, daily record data is written in distributed data base.Log processing equipment 12 is by distributed data
Data in library are synchronized to search server 13.Log query equipment 14 sends inquiry request to search server 13 and carries out log
Data query.
Fig. 2 is the flow diagram for the log processing method that one embodiment of the invention provides.Log provided in this embodiment
The executing subject of processing method is log processing equipment.As shown in Fig. 2, this method comprises:
S201, daily record data to be processed is received.
In the present embodiment, daily record data to be processed can be sent to log processing equipment by log generating device.Day
Will processing equipment receives the daily record data to be processed.
S202, distributed data base is written into the daily record data to be processed.
In the present embodiment, log processing to be processed can be written to distributed data base by log processing equipment.It can
Selection of land, distributed data base can be distributed relation database service (Distributed Relational Database
Service, DRDS), DRDS highly compatible MySQL agreement and grammer, support automatization level split, online smoothing it is scalable hold,
Resilient expansion, transparent read and write abruption have database Life cycle O&M control ability.Log processing equipment can be by inserting
Enter operation and daily record data to be processed is written to DRDS.
Whether the daily record data in S203, the monitoring distributed data base updates, wherein described update includes new
Increase the daily record data to be processed.
In the present embodiment, it may include increasing log to be processed newly that the daily record data in distributed data base, which occurs to update,
Data can also include being saved in daily record data to be deleted, edited, merged, be not limited thereto.Log processing equipment
It can monitor whether the daily record data in distributed data base updates.
When S204, the daily record data in the distributed data base update, by the distributed data base more
New daily record data is synchronized to search server, so that described search server searches the inquiry after receiving inquiry request
Request corresponding daily record data.
In the present embodiment, log processing equipment is when monitoring that the daily record data in distributed data base updates,
The daily record data updated in distributed data base is synchronized to search server.Search server is used to receive log generation
After equipment/log query equipment inquiry request, the corresponding daily record data of inquiry request is searched, generates query result, and to day
Will generating device/log query equipment returns to query result.Log generating device/log query equipment can show query result
Show and is checked to user.
The embodiment of the present invention is by receiving daily record data to be processed；Distributed data is written into daily record data to be processed
Library；Whether the daily record data in monitoring distributed data base updates, wherein updating includes increasing log number to be processed newly
According to；When daily record data in distributed data base updates, the daily record data updated in distributed data base is synchronized to
Search server, so that search server searches the corresponding daily record data of inquiry request, Neng Goutong after receiving inquiry request
It crosses and daily record data is synchronized to search server, so that the write-in and inquiry to daily record data are separated from each other, to improve to day
The response speed of the inquiry of will data.
Optionally, search server can be ES (Elasticsearch) search server, and ES search server is to be based on
The search server of Lucene, it provides the full-text search engine of a distributed multi-user ability, is based on RESTful web
Interface.Design pattern is distributed real-time files storage, and each field is indexed, and allows to be searched.?
Response speed aspect in face of big data quantity inquiry operation is particularly pertinent, can reach search in real time, stablize, is reliable, is quick
Inquiry.
Fig. 3 is the flow diagram for the log processing method that further embodiment of this invention provides.The present embodiment to receive to
The specific implementation process of the daily record data of processing is described in detail.As shown in figure 3, this method comprises:
Default log topic in S301, subscription Message Queuing system.
In the present embodiment, log generating device and log processing equipment carry out daily record data in a manner of message queue
Transmission.Log processing equipment subscribes to the default log topic in message queue first.Default log topic is to be used for transmission day master
The theme of data can specifically be set according to actual needs, is not limited thereto.
It whether there is log information to be processed under S302, the monitoring default log topic, wherein described to be processed
Log information includes the daily record data to be processed.
In the present embodiment, the message in message queue comprising daily record data to be processed is message to be processed.At log
Managing equipment monitoring whether there is the default corresponding log information to be processed of log topic.
Optionally, log information to be processed is that log generating device generates and sends in the Message Queuing system
Log information under the default log topic.
In the present embodiment, log generating device disappears the log for carrying daily record data after generating new daily record data
It ceases in the message queue sent and presets under log topic, disappear so that log processing equipment can receive this under default log topic
Breath.
S303, under monitoring the default log topic there are when the log information to be processed, from the message
The log information to be processed is obtained in queue system, and is extracted from the log information to be processed described to be processed
Daily record data.
In the present embodiment, there are log informations to be processed in the case where monitoring default log topic for log processing equipment
When, log information to be processed is obtained under log topic from presetting in Message Queuing system, and from log information to be processed
Extraction obtains daily record data to be processed.
Optionally, log generating device and log processing equipment carry out the transmission of daily record data by RocketMQ.Log
Generating device can send default log topic for log information by the RocketMQ manufacturing side.Log processing equipment can lead to
It crosses RocketMQ consumption terminal and subscribes to default log topic, monitor the message in default log topic, connect from default log topic
Message of the packet receiving containing daily record data.Wherein, RocketMQ is the Message Queuing Middleware of a kind of high-performance, high-throughput.Using
The mode of message queue, the business diary that the manufacturing side will record are sent in the message queue of corresponding theme by prescribed form,
It does not need to be concerned about whether consumed, system jams will not be caused when system is busy.Consumption terminal can supervise message queue
Control obtains the business diary message of corresponding theme at any time, is subsequently inserted into database.The loose coupling realized by RocketMQ
System availability and scalability can be improved in architecture design.
S304, distributed data base is written into the daily record data to be processed.
In the present embodiment, S304 is similar with the S202 in Fig. 2 embodiment, and details are not described herein again.
Whether the daily record data in S305, the monitoring distributed data base updates, wherein described update includes new
Increase the daily record data to be processed.
In the present embodiment, S305 is similar with the S203 in Fig. 2 embodiment, and details are not described herein again.
When S306, the daily record data in the distributed data base update, by the distributed data base more
New daily record data is synchronized to search server, so that described search server searches the inquiry after receiving inquiry request
Request corresponding daily record data.
In the present embodiment, S306 is similar with the S204 in Fig. 2 embodiment, and details are not described herein again.
Optionally, " daily record data updated in the distributed data base is synchronized to search server " in S306 can be with
Include:
By data transport service (Data Transmission Service, DTS) by the distributed data base more
Data after new are synchronized to described search server.
In the present embodiment, whether log processing equipment can monitor the daily record data in distributed data base by DTS
It updates, when the daily record data in distributed data base updates, day for will being updated in distributed data base by DTS
Will data are synchronized to search server.Wherein, DTS be used for data transmitted with it is synchronous, due to the high property of its framework itself
Can, highly reliable, high extension special efficacy, and can accomplish seamless connection with DRDS, in terms of its isomerism, support DRDS data are real
When be synchronized on ES search server.
Fig. 4 is the schematic diagram of a log processing process provided in an embodiment of the present invention.As shown in figure 4, electric business system is logical
It crosses the RocketMQ manufacturing side to send log information on log topic, RocketMQ consumption terminal receives the day under the log topic
Will message, obtains daily record data, and daily record data is written to DRDS database.DTS monitors the data in DRDS database in real time,
Daily record data in DRDS database is synchronized to ES search server.Electric business system or log query system are to ES search server
It sends request and carries out log query.
The embodiment of the present invention optimizes function by asynchronous message synchronization and read and write abruption technology, uses message queue mode
Asynchronous operation DRDS database, the message manufacturing side do not need be concerned about whether message is consumed and used, do not need to carry out database
Connection.The real-time monitoring information queue of consumption terminal carries out in-stockroom operation after obtaining log information again.DTS data transfer tool is real-time
The variation of monitoring data library mileage evidence, and by the real time data synchronization of variation to Elasticsearch search server.Utilize ES
The characteristic of itself is added index operation, fast and easy retrieval to data.When the page inquires business diary, java
Program accesses ES by api, and ES passes through itself index operation to data, and according to querying condition, rapidly locating is reduced anti-
Duration is answered, the response time is largely shortened.By realizing asynchronous and synchronous and read and write abruption technology to business diary, database
Write operation is only done, Elasticsearch does query and search operation, reduces database host disk, network pressure, and data pass through
DTS real-time synchronization guarantees the integrality of data.This programme uses message queue+DRDS+DTS+Elasticsearch framework, right
Message is written and carries out asynchronous and synchronous operation, retrieval log is written and read separation, substantially reduces the pressure of system, optimization many condition inspection
Rope inquiry velocity.
The present embodiment is by receiving daily record data to be processed；Distributed data base is written into daily record data to be processed；
Whether the daily record data in monitoring distributed data base updates, wherein updating includes increasing daily record data to be processed newly；?
When daily record data in distributed data base updates, the daily record data updated in distributed data base is synchronized to search clothes
It is engaged in device, it, can be by by day so that search server searches the corresponding daily record data of inquiry request after receiving inquiry request
Will data are synchronized to search server, so that the write-in and inquiry to daily record data are separated from each other, to improve to daily record data
Inquiry response speed.
Fig. 5 is the structural schematic diagram for the log processing device that one embodiment of the invention provides.As shown in figure 5, at the log
Managing device 50 includes: receiving module 501, writing module 502, monitoring modular 503 and synchronization module 504.
Receiving module 501, for receiving daily record data to be processed.
Writing module 502, for distributed data base to be written in the daily record data to be processed.
Whether monitoring modular 503, the daily record data for monitoring in the distributed data base update, wherein institute
Stating update includes increasing the daily record data to be processed newly.
Synchronization module 504, when being updated for the daily record data in the distributed data base, by the distribution
The daily record data updated in database is synchronized to search server, so that described search server is looked into after receiving inquiry request
Look for the corresponding daily record data of the inquiry request.
The embodiment of the present invention receives daily record data to be processed by receiving module；Writing module is by log number to be processed
According to write-in distributed data base；Whether the daily record data in monitoring module monitors distributed data base updates, wherein updates
Including increasing daily record data to be processed newly；When daily record data of the synchronization module in distributed data base updates, it will be distributed
The daily record data updated in formula database is synchronized to search server, so that search server is searched after receiving inquiry request
The corresponding daily record data of inquiry request, can be by being synchronized to search server for daily record data, so that writing to daily record data
Enter and inquiry is separated from each other, to improve the response speed of the inquiry to daily record data.
Fig. 6 is the structural schematic diagram for the log processing device that further embodiment of this invention provides.As shown in fig. 6, this implementation
It can also include: to order on the basis of the log processing device that the embodiment shown in Fig. 5 of log processing device 50 that example provides provides
Read module 505.
Optionally, the subscribing module 505 is used for:
Subscribe to the default log topic in Message Queuing system；
The receiving module 501 is used for:
It monitors under the default log topic with the presence or absence of log information to be processed, wherein the log to be processed
Message includes the daily record data to be processed；
Under monitoring the default log topic there are when the log information to be processed, from the message queue system
The log information to be processed is obtained in system, and the log number to be processed is extracted from the log information to be processed
According to.
Optionally, the log information to be processed is that log generating device generates and sends the Message Queuing system
In the default log topic under log information.
Optionally, the synchronization module 504 is used for:
Data updated in the distributed data base are synchronized to described search service by data transport service DTS
Device.
Log processing device provided in an embodiment of the present invention can be used for executing above-mentioned embodiment of the method, realization principle
Similar with technical effect, details are not described herein again for the present embodiment.
Fig. 7 is the hardware structural diagram for the log processing equipment that one embodiment of the invention provides.As shown in fig. 7, this reality
The log processing equipment 70 for applying example offer includes: at least one processor 701 and memory 702.The log processing equipment 70 is also
Including communication component 703.Wherein, processor 701, memory 702 and communication component 703 are connected by bus 704.
During specific implementation, at least one processor 701 executes the computer execution that the memory 702 stores and refers to
It enables, so that at least one processor 701 executes log processing method as above.
The specific implementation process of processor 701 can be found in above method embodiment, and it is similar that the realization principle and technical effect are similar,
Details are not described herein again for the present embodiment.
In above-mentioned embodiment shown in Fig. 7, it should be appreciated that processor can be central processing unit (English:
Central Processing Unit, referred to as: CPU), can also be other general processors, digital signal processor (English:
Digital Signal Processor, referred to as: DSP), specific integrated circuit (English: Application Specific
Integrated Circuit, referred to as: ASIC) etc..General processor can be microprocessor or the processor is also possible to
Any conventional processor etc..Hardware processor can be embodied directly in conjunction with the step of invention disclosed method to have executed
At, or in processor hardware and software module combination execute completion.
Memory may include high speed RAM memory, it is also possible to and it further include non-volatile memories NVM, for example, at least one
Magnetic disk storage.
Bus can be industry standard architecture (Industry Standard Architecture, ISA) bus, outer
Portion's apparatus interconnection (Peripheral Component, PCI) bus or extended industry-standard architecture (Extended
Industry Standard Architecture, EISA) bus etc..Bus can be divided into address bus, data/address bus, control
Bus etc..For convenient for indicating, the bus in illustrations does not limit only a bus or a type of bus.
The application also provides a kind of log processing system, including log processing equipment as described above and search server；
Described search server after receiving inquiry request for searching the corresponding daily record data of the inquiry request.
Optionally, which further includes log generating device and log query equipment；
The log generating device is sent for generating log information to be processed, and by the log information to be processed
Under default log topic into the Message Queuing system, wherein the log information to be processed includes day to be processed
Will data；
The log query equipment is used to receive the inquiry instruction of user's input, is generated according to the inquiry instruction corresponding
Inquiry request, and described search server is sent by the inquiry request, receiving looking into for described search server return
After asking result, the query result is shown.
Log processing system provided in an embodiment of the present invention, implementing principle and technical effect and above-mentioned mode embodiment
Similar, details are not described herein again for the present embodiment.
The application also provides a kind of computer readable storage medium, and calculating is stored in the computer readable storage medium
Machine executes instruction, and when processor executes the computer executed instructions, realizes log processing method as above.
Above-mentioned computer readable storage medium, above-mentioned readable storage medium storing program for executing can be by any kind of volatibility or non-
Volatile storage devices or their combination realize that, such as static random access memory (SRAM), electrically erasable is only
It reads memory (EEPROM), Erasable Programmable Read Only Memory EPROM (EPROM), programmable read only memory (PROM) is read-only to deposit
Reservoir (ROM), magnetic memory, flash memory, disk or CD.Readable storage medium storing program for executing can be general or specialized computer capacity
Any usable medium enough accessed.
A kind of illustrative readable storage medium storing program for executing is coupled to processor, to enable a processor to from the readable storage medium storing program for executing
Information is read, and information can be written to the readable storage medium storing program for executing.Certainly, readable storage medium storing program for executing is also possible to the composition portion of processor
Point.Processor and readable storage medium storing program for executing can be located at specific integrated circuit (Application Specific Integrated
Circuits, referred to as: ASIC) in.Certainly, processor and readable storage medium storing program for executing can also be used as discrete assembly and be present in equipment
In.
Those of ordinary skill in the art will appreciate that: realize that all or part of the steps of above-mentioned each method embodiment can lead to
The relevant hardware of program instruction is crossed to complete.Program above-mentioned can be stored in a computer readable storage medium.The journey
When being executed, execution includes the steps that above-mentioned each method embodiment to sequence；And storage medium above-mentioned include: ROM, RAM, magnetic disk or
The various media that can store program code such as person's CD.
Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention., rather than its limitations；To the greatest extent
Pipe present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: its according to
So be possible to modify the technical solutions described in the foregoing embodiments, or to some or all of the technical features into
Row equivalent replacement；And these are modified or replaceed, various embodiments of the present invention technology that it does not separate the essence of the corresponding technical solution
The range of scheme.
Claims (12)
1. a kind of log processing method characterized by comprising
Receive daily record data to be processed；
Distributed data base is written into the daily record data to be processed；
Monitor whether the daily record data in the distributed data base updates, wherein it is described update include it is newly-increased described to
The daily record data of processing；
When daily record data in the distributed data base updates, the log number that will be updated in the distributed data base
According to being synchronized to search server, so as to search the inquiry request after receiving inquiry request corresponding for described search server
Daily record data.
2. the method according to claim 1, wherein also being wrapped before reception daily record data to be processed
It includes:
Subscribe to the default log topic in Message Queuing system；
It is described to receive daily record data to be processed, comprising:
It monitors under the default log topic with the presence or absence of log information to be processed, wherein the log information to be processed
Include the daily record data to be processed；
Under monitoring the default log topic there are when the log information to be processed, from the Message Queuing system
The log information to be processed is obtained, and extracts the daily record data to be processed from the log information to be processed.
3. according to the method described in claim 2, it is characterized in that, the log information to be processed is raw for log generating device
At and the log information that is sent under the default log topic in the Message Queuing system.
4. method according to any one of claims 1 to 3, which is characterized in that it is described by the distributed data base more
Data after new are synchronized to search server, comprising:
Data updated in the distributed data base are synchronized to described search server by data transport service DTS.
5. a kind of log processing device characterized by comprising
Receiving module, for receiving daily record data to be processed；
Writing module, for distributed data base to be written in the daily record data to be processed；
Whether monitoring modular, the daily record data for monitoring in the distributed data base update, wherein the update packet
It includes and increases the daily record data to be processed newly；
Synchronization module, when being updated for the daily record data in the distributed data base, by the distributed data base
The daily record data of middle update is synchronized to search server, so that described in described search server searches after receiving inquiry request
The corresponding daily record data of inquiry request.
6. device according to claim 5, which is characterized in that further include subscribing module, the subscribing module is used for:
Subscribe to the default log topic in Message Queuing system；
The receiving module is used for:
It monitors under the default log topic with the presence or absence of log information to be processed, wherein the log information to be processed
Include the daily record data to be processed；
Under monitoring the default log topic there are when the log information to be processed, from the Message Queuing system
The log information to be processed is obtained, and extracts the daily record data to be processed from the log information to be processed.
7. device according to claim 6, which is characterized in that the log information to be processed is raw for log generating device
At and the log information that is sent under the default log topic in the Message Queuing system.
8. according to the described in any item devices of claim 5 to 7, which is characterized in that the synchronization module is used for:
Data updated in the distributed data base are synchronized to described search server by data transport service DTS.
9. a kind of log processing equipment characterized by comprising at least one processor and memory；
The memory stores computer executed instructions；
At least one described processor executes the computer executed instructions of the memory storage, so that at least one described processing
Device executes such as the described in any item log processing methods of Claims 1-4.
10. a kind of log processing system, which is characterized in that including search server and log processing as claimed in claim 9
Equipment；
Described search server after receiving inquiry request for searching the corresponding daily record data of the inquiry request.
11. system according to claim 10, which is characterized in that further include log generating device and log query equipment；
The log generating device sends institute for the log information to be processed for generating log information to be processed
It states under the default log topic in Message Queuing system, wherein the log information to be processed includes log number to be processed
According to；
The log query equipment is used to receive the inquiry instruction of user's input, generates corresponding inquiry according to the inquiry instruction
Request, and described search server is sent by the inquiry request, in the inquiry knot for receiving the return of described search server
After fruit, the query result is shown.
12. a kind of computer readable storage medium, which is characterized in that be stored with computer in the computer readable storage medium
It executes instruction, when processor executes the computer executed instructions, realizes such as the described in any item logs of Claims 1-4
Processing method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910427734.0A CN110134648A (en) | 2019-05-22 | 2019-05-22 | Log processing method, device, equipment, system and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910427734.0A CN110134648A (en) | 2019-05-22 | 2019-05-22 | Log processing method, device, equipment, system and computer readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110134648A true CN110134648A (en) | 2019-08-16 |
Family
ID=67572124
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910427734.0A Pending CN110134648A (en) | 2019-05-22 | 2019-05-22 | Log processing method, device, equipment, system and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110134648A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110765091A (en) * | 2019-09-09 | 2020-02-07 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Account checking method and system |
| CN111177081A (en) * | 2019-12-31 | 2020-05-19 | 广州极尚网络技术有限公司 | Game log content query method and device, computer equipment and storage medium |
| CN111367692A (en) * | 2020-03-09 | 2020-07-03 | 政采云有限公司 | Search engine data processing method and device, electronic equipment and medium |
| CN111400368A (en) * | 2020-03-06 | 2020-07-10 | 网宿科技股份有限公司 | Log searching method and device of distributed server system |
| CN111506646A (en) * | 2020-03-16 | 2020-08-07 | 阿里巴巴集团控股有限公司 | Data synchronization method, device, system, storage medium and processor |
| CN111651481A (en) * | 2020-05-29 | 2020-09-11 | 泰康保险集团股份有限公司 | Log processing method, device, equipment and computer readable storage medium |
| CN111752906A (en) * | 2019-12-30 | 2020-10-09 | 北京沃东天骏信息技术有限公司 | Request response method, device and system, computer system and readable storage medium |
| CN111753015A (en) * | 2020-06-28 | 2020-10-09 | 中国银行股份有限公司 | Data query method and device of payment clearing system |
| CN112052247A (en) * | 2020-09-29 | 2020-12-08 | 微医云(杭州)控股有限公司 | Index updating system, method and device of search engine, electronic equipment and storage medium |
| CN112084264A (en) * | 2020-09-11 | 2020-12-15 | 北京金山云网络技术有限公司 | Log data synchronization method and device and server |
| CN112231296A (en) * | 2020-09-30 | 2021-01-15 | 北京金山云网络技术有限公司 | Distributed log processing method, device, system, equipment and medium |
| CN113360554A (en) * | 2020-03-06 | 2021-09-07 | 深圳法大大网络科技有限公司 | Method and equipment for extracting, converting and loading ETL (extract transform load) data |
| CN113722401A (en) * | 2021-11-04 | 2021-11-30 | 树根互联股份有限公司 | Data caching method and device, computer equipment and readable storage medium |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617176A (en) * | 2013-11-04 | 2014-03-05 | 广东电子工业研究院有限公司 | Method for achieving automatic synchronization of multi-source heterogeneous data resources |
| CN103856353A (en) * | 2014-03-06 | 2014-06-11 | 上海爱数软件有限公司 | Service log data access and statistic analysis method and device |
| CN104486107A (en) * | 2014-12-05 | 2015-04-01 | 曙光信息产业(北京)有限公司 | Log collection device and method |
| US20150378774A1 (en) * | 2014-06-26 | 2015-12-31 | Amazon Technologies, Inc. | Log-based concurrency control using signatures |
| CN107402963A (en) * | 2017-06-20 | 2017-11-28 | 阿里巴巴集团控股有限公司 | Search for construction method, the method for pushing and device and equipment of incremental data of data |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN108920659A (en) * | 2018-07-03 | 2018-11-30 | 广州唯品会信息科技有限公司 | Data processing system and its data processing method, computer readable storage medium |
| CN109063066A (en) * | 2018-07-20 | 2018-12-21 | 中国邮政储蓄银行股份有限公司 | Data query method and device, data management system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
-
2019
- 2019-05-22 CN CN201910427734.0A patent/CN110134648A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103617176A (en) * | 2013-11-04 | 2014-03-05 | 广东电子工业研究院有限公司 | Method for achieving automatic synchronization of multi-source heterogeneous data resources |
| CN103856353A (en) * | 2014-03-06 | 2014-06-11 | 上海爱数软件有限公司 | Service log data access and statistic analysis method and device |
| US20150378774A1 (en) * | 2014-06-26 | 2015-12-31 | Amazon Technologies, Inc. | Log-based concurrency control using signatures |
| CN104486107A (en) * | 2014-12-05 | 2015-04-01 | 曙光信息产业(北京)有限公司 | Log collection device and method |
| CN107783975A (en) * | 2016-08-24 | 2018-03-09 | 北京京东尚科信息技术有限公司 | The method and apparatus of distributed data base synchronization process |
| CN107402963A (en) * | 2017-06-20 | 2017-11-28 | 阿里巴巴集团控股有限公司 | Search for construction method, the method for pushing and device and equipment of incremental data of data |
| CN108920659A (en) * | 2018-07-03 | 2018-11-30 | 广州唯品会信息科技有限公司 | Data processing system and its data processing method, computer readable storage medium |
| CN109063066A (en) * | 2018-07-20 | 2018-12-21 | 中国邮政储蓄银行股份有限公司 | Data query method and device, data management system |
| CN109753531A (en) * | 2018-12-26 | 2019-05-14 | 深圳市麦谷科技有限公司 | A kind of big data statistical method, system, computer equipment and storage medium |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110765091A (en) * | 2019-09-09 | 2020-02-07 | 上海陆家嘴国际金融资产交易市场股份有限公司 | Account checking method and system |
| CN111752906B (en) * | 2019-12-30 | 2023-12-05 | 北京沃东天骏信息技术有限公司 | Request response method, apparatus and system, computer system and readable storage medium |
| CN111752906A (en) * | 2019-12-30 | 2020-10-09 | 北京沃东天骏信息技术有限公司 | Request response method, device and system, computer system and readable storage medium |
| CN111177081B (en) * | 2019-12-31 | 2023-11-14 | 广州极尚网络技术有限公司 | Game log content query method, device, computer equipment and storage medium |
| CN111177081A (en) * | 2019-12-31 | 2020-05-19 | 广州极尚网络技术有限公司 | Game log content query method and device, computer equipment and storage medium |
| CN111400368A (en) * | 2020-03-06 | 2020-07-10 | 网宿科技股份有限公司 | Log searching method and device of distributed server system |
| CN111400368B (en) * | 2020-03-06 | 2023-12-12 | 网宿科技股份有限公司 | Log searching method and device of distributed server system |
| CN113360554A (en) * | 2020-03-06 | 2021-09-07 | 深圳法大大网络科技有限公司 | Method and equipment for extracting, converting and loading ETL (extract transform load) data |
| CN113360554B (en) * | 2020-03-06 | 2023-06-23 | 深圳法大大网络科技有限公司 | Method and equipment for extracting, converting and loading ETL (extract transform load) data |
| CN111367692A (en) * | 2020-03-09 | 2020-07-03 | 政采云有限公司 | Search engine data processing method and device, electronic equipment and medium |
| CN111367692B (en) * | 2020-03-09 | 2023-08-22 | 政采云有限公司 | Search engine data processing method and device, electronic equipment and medium |
| CN111506646A (en) * | 2020-03-16 | 2020-08-07 | 阿里巴巴集团控股有限公司 | Data synchronization method, device, system, storage medium and processor |
| CN111506646B (en) * | 2020-03-16 | 2023-05-02 | 阿里巴巴集团控股有限公司 | Data synchronization method, device, system, storage medium and processor |
| CN111651481A (en) * | 2020-05-29 | 2020-09-11 | 泰康保险集团股份有限公司 | Log processing method, device, equipment and computer readable storage medium |
| CN111651481B (en) * | 2020-05-29 | 2023-08-22 | 泰康保险集团股份有限公司 | Log processing method, device, equipment and computer readable storage medium |
| CN111753015B (en) * | 2020-06-28 | 2023-09-19 | 中国银行股份有限公司 | Data query method and device of payment clearing system |
| CN111753015A (en) * | 2020-06-28 | 2020-10-09 | 中国银行股份有限公司 | Data query method and device of payment clearing system |
| CN112084264A (en) * | 2020-09-11 | 2020-12-15 | 北京金山云网络技术有限公司 | Log data synchronization method and device and server |
| CN112052247A (en) * | 2020-09-29 | 2020-12-08 | 微医云(杭州)控股有限公司 | Index updating system, method and device of search engine, electronic equipment and storage medium |
| CN112231296A (en) * | 2020-09-30 | 2021-01-15 | 北京金山云网络技术有限公司 | Distributed log processing method, device, system, equipment and medium |
| CN113722401A (en) * | 2021-11-04 | 2021-11-30 | 树根互联股份有限公司 | Data caching method and device, computer equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110134648A (en) | Log processing method, device, equipment, system and computer readable storage medium | |
| CN110321387B (en) | Data synchronization method, equipment and terminal equipment | |
| CN102142032B (en) | Method and system for reading and writing data of distributed file system | |
| CN109977129A (en) | Multi-stage data caching method and equipment | |
| CN111258978B (en) | Data storage method | |
| US11048423B2 (en) | Method and system for synchronizing requests related to key-value storage having different portions | |
| CN110347651A (en) | Method of data synchronization, device, equipment and storage medium based on cloud storage | |
| KR101545626B1 (en) | System for interoperation between dds and dbms | |
| CN109787884B (en) | Message pushing method and device | |
| CN111610979A (en) | API gateway subjected to persistence and coupling degree optimization and method thereof | |
| CN110955857A (en) | Service processing method and device for high concurrency environment | |
| CN110334145A (en) | The method and apparatus of data processing | |
| CN108564462A (en) | Acquisition methods, terminal device and the medium of collage-credit data | |
| CN110781149A (en) | Method, device, equipment and storage medium for managing live broadcast room information | |
| CN110175152A (en) | A kind of log inquiring method, transfer server cluster and log query system | |
| CN104468399A (en) | Data transmission method and device and server | |
| CN105426125B (en) | A kind of date storage method and device | |
| CN113256355B (en) | Method, device, medium, equipment and system for determining integral rights and interests in real time | |
| CN113051271B (en) | Cold and hot data separation method, device and equipment thereof | |
| CN113051244B (en) | Data access method and device, and data acquisition method and device | |
| CN110032608B (en) | System data information assembling method and device, storage medium and electronic equipment | |
| CN111447267B (en) | Information synchronization method, device, computer readable storage medium and equipment | |
| CN111399753B (en) | Method and device for writing pictures | |
| CN112783925B (en) | Paging retrieval method and device | |
| CN106484379B (en) | A kind of processing method and processing device of application |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190816 |
|
| RJ01 | Rejection of invention patent application after publication |