CN109885545A - It stores, the method, apparatus of inquiry log information - Google Patents
It stores, the method, apparatus of inquiry log information Download PDFInfo
- Publication number
- CN109885545A CN109885545A CN201910106607.0A CN201910106607A CN109885545A CN 109885545 A CN109885545 A CN 109885545A CN 201910106607 A CN201910106607 A CN 201910106607A CN 109885545 A CN109885545 A CN 109885545A
- Authority
- CN
- China
- Prior art keywords
- log information
- template
- log
- database
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
This application provides a kind of methods for storing log information, this method comprises: identifying first part and the second part of a plurality of log information from a plurality of log information, the first part of a plurality of log information is to be less than or equal to identical part in first threshold log information in a plurality of log information, and the second part of a plurality of log information is to be less than or equal to different part in second threshold log information in a plurality of log information；The second part that a plurality of log information is replaced with occupy-place identifier forms the corresponding log information template of a plurality of log information；The second part of the corresponding log information template of a plurality of log information and a plurality of log information is stored respectively to database.Technical solution provided by the present application can reduce the identical occupied memory space in part in a plurality of log information, save storage resource.
Description
Technical field
This application involves field of storage, and believe more particularly, to a kind of method for storing log information, inquiry log
The method, apparatus and computer readable storage medium of breath.
Background technique
Computer log is in computer system operational process by operating system, middleware, platform itself generate or
It is generated by the program assembly of User Exploitation, has recorded the operating status and service condition of equipment and system itself, these descriptions
Information essentially describes the key operation that system is carried out and the mistake that system is occurred in the process of running and exception etc..With
Family can be by the analysis to system log, and understanding system often occurs in operation or the accidental problem, so as to
Targetedly to improve the operation maintenance to system, and then improve the safety and efficiency of system operation.
In traditional log information storage scheme, original log information is stored directly in database.On the one hand, it passes
There are a large amount of duplicate information in original log in the log information storage scheme of system, is accounted for so as to cause a large amount of duplicate information
With most of storage resource, the waste in resource is caused.On the other hand, it is retrieved in log information of the user to storage
During, a large amount of duplicate information can make recall precision low in original log, and user is a large amount of to what is retrieved
Original log when carrying out data analysis, a large amount of duplicate parts can make the real-time of customer analysis process poor, be unfavorable for
It is quickly found the problem by log.
Summary of the invention
The application provides a kind of method of method and inquiry log information for storing log information, can reduce a plurality of log
The duplicate occupied memory space in part in information saves storage resource, and shortens user to the retrieval time of log information,
Improve the use value of log information.
In a first aspect, a kind of method for storing log information is provided, this method comprises: identifying from a plurality of log information
The first part of a plurality of log information and second part out replace the second of a plurality of log information with occupy-place identifier
Part forms the corresponding log information template of a plurality of log information, and the log information template includes a plurality of log
The first part of information, by second of a plurality of corresponding log information template of log information and a plurality of log information
Divide and is stored respectively to database.
Wherein, the first part of a plurality of log information is to be less than or equal to the first threshold in a plurality of log information
It is worth identical part in a log information, the second part of a plurality of log information is to be less than in a plurality of log information
Or it is equal to different part in second threshold log information.
It should be understood that first threshold can be less than or equal to a plurality of log information total quantity, second threshold can be less than or
Equal to the total quantity of a plurality of log information, first threshold and second threshold can be equal or unequal.
In above-mentioned technical proposal, original log information can be handled, in a plurality of log information, from being less than or wait
Identical part and different part are identified in a plurality of log information of a plurality of log information quantity.A plurality of log is believed
Identical part only stores one in breath, so that the log information after simplifying does not have to carry in log information template one
Long string character string, reduces memory space shared by log information, saves the storage resource consumption in log service system.
In one possible implementation, the method also includes: it is corresponding described to generate a plurality of log information
The mark of log information template.
In above-mentioned technical proposal, unique mark corresponding to each log information template can be calculated, so as to
To be gone back by log information template identification to log information during the user in later period retrieves log information
Original, so that user is available to arrive original log information.Not only section can be understood by the analysis to original log information
Point often occurs in operation or the accidental problem, can also pass through mark corresponding to analysis log information template
The regularity of distribution excavates potential system exception, and issues warning reminding in advance.
In alternatively possible implementation, user examines by log information of the log services API to storage
During rope, the search of keyword can be carried out in log information template.
In above-mentioned technical proposal, the log information that user needs is carried out by passing through keyword in log information template
Search, since log information template will not be repeated largely, the log information of user's needs can be quickly found, is reduced
The time of search improves the real-time of customer analysis process, is conducive to quickly find the problem by log.
In alternatively possible implementation, by the second part of a plurality of log information and the log information mould
The mark of plate is stored in the first space of the database；By the mark of the log information template and the log information template
It is stored in the second space of the database.
Second aspect provides a kind of method of inquiry log information, and the method is applied to log information inquiry system,
The log information inquiry system includes database, and the database purchase has the second part of a plurality of log information and described more
The corresponding log information template of log information, the log information template includes the first part of a plurality of log information,
The first part of a plurality of log information is to be less than or equal to first threshold log information in a plurality of log information
In identical part, the second part of a plurality of log information is to be less than or equal to the second threshold in a plurality of log information
It is worth different part in a log information, which comprises
Inquiry request is received, the inquiry request is for inquiring the log information stored in database；According to the inquiry
Request obtains the second part of a plurality of log information of storage from the database and a plurality of log information corresponds to
Log information template；Bring the second part of a plurality of log information into a plurality of log information corresponding log information
Template obtains corresponding log information, returns to the corresponding log information to the client for issuing the inquiry request,
In above-mentioned technical proposal into, the second part of a plurality of log information can be brought to the corresponding log of a plurality of log information
The original log information of formation is back to user by information model.User can be made to the original log information got
It is analyzed, understanding node often occurs in operation or the accidental problem.
In one possible implementation, before the reception inquiry request, the method also includes: from a plurality of day
Identified in will information a plurality of log information the first part and the second part；
The second part of a plurality of log information is replaced with occupy-place identifier, and it is corresponding to form a plurality of log information
Log information template；
The second part of a plurality of corresponding log information template of log information and a plurality of log information is distinguished
It stores to the database.
In alternatively possible implementation, the method also includes: generate the corresponding institute of a plurality of log information
State the mark of log information template.
In alternatively possible implementation, the method also includes: by the second part of a plurality of log information
The first space of the database is stored in the mark of the log information template；By the log information template and the day
The mark of will information model is stored in the second space of the database.
The third aspect, provides a kind of device for storing log information, which includes:
Identification module, for identified from a plurality of log information a plurality of log information first part and second
Point, wherein the first part of a plurality of log information is to be less than or equal to first threshold item in a plurality of log information
Identical part in log information, the second part of a plurality of log information are to be less than or wait in a plurality of log information
The different part in second threshold log information；
Processing module is formed described more for replacing the second part of a plurality of log information by occupy-place identifier
The corresponding log information template of log information, the log information template includes the first part of a plurality of log information；
Memory module, for by a plurality of corresponding log information template of log information and a plurality of log information
Second part is stored respectively to database.
In one possible implementation, described device further include: generation module, for generating a plurality of log letter
Cease the mark of the corresponding log information template.
In alternatively possible implementation, the memory module is specifically used for: by the of a plurality of log information
The mark of two parts and the log information template is stored in the first space of the database；By the log information template and
The mark of the log information template is stored in the second space of the database.
Fourth aspect provides a kind of device of inquiry log information, and described device is applied to log information inquiry system,
The log information inquiry system includes database, and the database purchase has the second part of a plurality of log information and described more
The corresponding log information template of log information, the log information template includes the first part of a plurality of log information,
The first part of a plurality of log information is to be less than or equal to first threshold log information in a plurality of log information
In identical part, the second part of a plurality of log information is to be less than or equal to the second threshold in a plurality of log information
It is worth different part in a log information, described device includes:
Receiving module, for receiving inquiry request, the inquiry request is for inquiring the log information stored in database；
Module is obtained, for obtaining a plurality of log information of storage from the database according to the inquiry request
Second part and the corresponding log information template of a plurality of log information；It is also used to the second of a plurality of log information
Part brings the corresponding log information template of a plurality of log information into and obtains corresponding log information.
Optionally, the device of the inquiry log information can also include return module, for asking to the sending inquiry
The client asked returns to the corresponding log information
In one possible implementation, described device further include: identification module, for knowing from a plurality of log information
Not Chu a plurality of log information the first part and the second part；
Processing module is formed described more for replacing the second part of a plurality of log information by occupy-place identifier
The corresponding log information template of log information；
Memory module, for by a plurality of corresponding log information template of log information and a plurality of log information
Second part is stored respectively to database.
In alternatively possible implementation, described device further include: generation module, for generating a plurality of log
The mark of the corresponding log information template of information.
In alternatively possible implementation, memory module is specifically used for: by second of a plurality of log information
Divide the first space that the database is stored in the mark of the log information template；By the log information template and described
The mark of log information template is stored in the second space of the database.
5th aspect provides a kind of device for storing log information.The device for storing log information includes at least one
Calculate node, each calculate node include processor and memory.The processor of this at least one calculate node is deposited for executing
The program code stored in reservoir, to execute the method in above-mentioned first aspect or the possible implementation of first aspect.
6th aspect, provides a kind of device of inquiry log information.The device of inquiry log information includes at least one
Calculate node, each calculate node include processor and memory.The processor of this at least one calculate node is deposited for executing
The program code stored in reservoir, to execute the method in above-mentioned second aspect or the possible implementation of second aspect.
7th aspect, the application provide a kind of non-transient non-volatile computer readable storage medium storing program for executing, which can
It reads to be stored with instruction in storage medium, when it runs at least one calculate node, so that at least one calculate node is held
Method in the above-mentioned first aspect of row or the possible implementation of first aspect.
Eighth aspect, the application provide a kind of non-transient non-volatile computer readable storage medium storing program for executing, which can
It reads to be stored with instruction in storage medium, when it runs at least one calculate node, so that at least one calculate node is held
Method in the above-mentioned second aspect of row or the possible implementation of second aspect.
9th aspect, this application provides a kind of computer program products comprising instruction, when it is at least one calculating
When being run on node, so that at least one calculate node executes in above-mentioned first aspect or the possible implementation of first aspect
Method.
Tenth aspect, this application provides a kind of computer program products comprising instruction, when it is at least one calculating
When being run on node, so that at least one calculate node executes in above-mentioned first aspect or the possible implementation of first aspect
Method.
The application can also be further combined on the basis of the implementation that above-mentioned various aspects provide to provide more
More implementations.
Detailed description of the invention
Fig. 1 is a kind of schematic block diagram of Distributed Log Server System 100 applied to the embodiment of the present application.
Fig. 2 is a kind of schematic flow chart of method for storing log information provided by the embodiments of the present application.
Fig. 3 is a kind of structural schematic figure of device 300 for storing log information provided by the embodiments of the present application.
Fig. 4 is a kind of structural schematic figure of device 400 for storing log information provided by the embodiments of the present application.
Fig. 5 is a kind of structural schematic figure of the device 500 of inquiry log information provided by the embodiments of the present application.
Fig. 6 is a kind of structural schematic figure of the device 600 of inquiry log information provided by the embodiments of the present application.
Specific embodiment
Below in conjunction with attached drawing, the technical solution in the application is described.
Computer log is in computer system operational process by operating system, middleware, platform itself generate or
It is generated by the program assembly of User Exploitation, has recorded the operating status and service condition of equipment and system itself, these descriptions
Information essentially describes the key operation that system is carried out and the mistake that system is occurred in the process of running and exception etc..With
Family can be by the analysis to system log, and understanding system often occurs in operation or the accidental problem, so as to
Targetedly to improve the operation maintenance to system, and then improve the safety and efficiency of system operation.
Under large-scale distributed scene, needs special log system or log services are responsible for collecting, store and
From the log information of a large amount of Managed nodes, and provide the log number that relevant Retrieval Interface analyzes their care for user filtering
According to.Distributed log system 100 is described in detail below.
Fig. 1 is a kind of schematic block diagram of Distributed Log Server System 100 applied to the embodiment of the present application.Such as Fig. 1
Shown, system 100 may include server-side 110, at least one node.
The embodiment of the present application is not specifically limited the quantity of at least one node, and for ease of description, the application is implemented
Example is described by taking node 120, node 130 as an example.
It should be understood that node 120, node 130 can be the different virtual machine (virtual installed on a physical host
Machine, VM), it is also possible to the VM installed on different physical hosts.Node 120, node 130 are also possible to physical host.
At least one node in log service system 100 in the process of running, operating system, platform itself or by
The program assembly of User Exploitation can produce corresponding log information, which describes what at least one node was carried out
Key operation and the mistake occurred in the process of running and exception etc..At least one node can be by the log information of generation
It is stored in corresponding journal file.User is by the analysis to system log, not only it will be seen that at least one node is being transported
In row often occur or the accidental problem, to targetedly change to the operation maintenance of at least one node
It is kind.Potential system exception can also be excavated, and in advance by the regularity of distribution of mark corresponding to analysis log information template
Issue warning reminding.
Log collection agency 121, log collection agency 131 are deployed on node 120 and node 130 respectively.In node 120
Log collection act on behalf of the log information saved in the journal file in 121 available nodes 120 under specified directory, and pass through
The log information for the node 120 that message-oriented middleware 114 will acquire reports to server-side 110.Likewise, the day in node 130
The log information saved in journal file in the available node 130 of will Collection agent 131 under specified directory, and pass through message
The log information for the node 130 that middleware 114 will acquire reports to server-side 110.
It should be noted that log collection proxy module 121, log collection proxy module 131 can be software program reality
Existing module.
Data preprocessing module 113 in log service system 110 can receive node 120 and/or node 130 divides
After the log information not reported, the log information reported can be pre-processed.And the log information after processing is deposited
Storage is in the distributed data base 112 in server-side 110.
Specifically, the data preprocessing module 113 in server-side 110 can receive what message-oriented middleware 114 reported
After the log information of at least one node, log information is stored into distributed data base 112 later plus label information.
In order to which user is when retrieving relevant log information, relevant log information can be scanned for by label information.
It should be understood that label information can be day self-contained by log information, to be also possible at least one node
Will Collection agent is got according to the attribute of the node where log information and the application service operated above.The application is implemented
Example is not specifically limited the content of label information, can include but is not limited to: the preservation of the journal file where log information
Path, address Internet protocol (internet protocol, IP) of node where log information, the affiliated service of log information
Region (region) information of the public cloud at place, information on services belonging to log information is (for example, service identifiers
(identifier, ID)), module information (for example, component ID) belonging to log information, tenant ID etc..
Log services application programming interfaces (application programinterface, API) in server-side 110
111 can provide user search function, and user can service API 111 by browser or command-line tool, usage log
The interface of offer is interacted with log service system 100.
Specifically, user can scan for the log information that it is needed by log services API 111, for example, can
With the various label informations and log keyword to log information, the log letter being stored in distributed data base 112 is obtained
Breath.In order to which according to the log information got, understanding node often occurs in operation or the accidental problem, Ke Yiyou
Targetedly the operation of node is safeguarded, to improve the running efficiency of system and efficiency of node.
In traditional log storage scheme, due to being that directly original log information is stored directly in database,
It is middle to store a large amount of duplicate information.On the one hand, after log information quantity is more than to a certain degree, a large amount of log informations are adopted
Collection, the transmission of cross-node, the occupancy of the resource in mass data library will all face huge challenge.On the other hand, original log is believed
The direct storage of breath can also reduce the service efficiency to the log information.For example, in the case where searching for scene generally, in flood tide original day
It is very long that time-consuming is searched in will information, when usage log information does abnormality detection and other relevant data mining analysis, is needed
It concentrates and a large amount of log informations is analyzed, real-time is poor, is unfavorable for quickly finding the problem.
The embodiment of the present application provides a kind of method of storing data, can handle original log information, leads to
A large amount of duplicate data in elimination initial data are crossed, memory space shared by original data is reduced.
It should be noted that Fig. 1 is a kind of possible scene applied to the embodiment of the present application, the embodiment of the present application is provided
Method be also applied in the scene that a large amount of duplicate non-structured data are stored.
Fig. 2 is a kind of schematic flow chart of method for storing log information provided by the embodiments of the present application.Such as Fig. 1 institute
Show, this method may include step 210-230, and step 210-230 is described in detail separately below.
Step 210: first part and the second part of a plurality of log information are identified from a plurality of log information.
The first part and second of a plurality of log information can be identified in the embodiment of the present application from a plurality of log information
Part, wherein first part, which can be, is less than or equal to identical portion in first threshold log information in a plurality of log information
Point, second part is to be less than or equal to different part in second threshold log information in a plurality of log information.
It should be understood that first threshold can be less than or equal to a plurality of log information total quantity, second threshold can be less than or
Equal to the total quantity of a plurality of log information.
It should be noted that first threshold and second threshold can be equal or unequal, the embodiment of the present application is to this
It is not especially limited.
Referring to Fig. 1, the log collection that can be at least one node in the embodiment of the present application is acted on behalf of to a plurality of log information
In be less than or equal to first threshold log information compare identification, the data prediction mould being also possible in server-side 110
Block 113 compares identification less than or equal to first threshold log information in a plurality of log information.As an example, exist
The processor and/or memory of log collection agency 121 in node 120 feelings more sufficient for the data volume of its processing
Under condition, after log collection agency 121 can obtain the original log information of storage in the journal file from specified directory,
It is identified to first threshold log information is less than or equal in a plurality of log information, identifies above-mentioned first part and second
Part.As another example, the processor and/or memory of the log collection agency 121 in node 120 is relative to its processing
Data volume for it is relatively limited in the case where, the data preprocessing module 113 in server-side 110 can receive at least one
After multiple original log informations for reporting of log collection agency in node, to being less than or equal to the in a plurality of log information
One threshold value log information compares identification, is less than or equal to first threshold log letter in a plurality of log information by comparing
Breath, identifies above-mentioned first part and second part.
Step 220: replacing the second part of a plurality of log information with occupy-place identifier, form a plurality of log information pair
The log information template answered.
The embodiment of the present application can be handled from the second part of the above-mentioned a plurality of log information identified, example
It such as, can be by replacing the second part in a plurality of log information with occupy-place identifier, so that it is corresponding to form an a plurality of log
One log information template, so that being the corresponding relationship of 1:M between log information template and M original log information, to disappear
Except duplicate parts a large amount of in a plurality of log information.Wherein, M is the positive integer greater than 1.
The type in the second part of a plurality of log information can be distinguished in the embodiment of the present application by occupy-place identifier,
The type for distinguishing changing unit in the second part of a plurality of log information can not be had to, the application is not especially limited this.Make
For an example, when types of variables in the second part for distinguishing a plurality of log information by occupy-place identifier, %d can be used
As the occupy-place identifier for indicating numeric type variable, use %s as the variable occupy-place identifier for indicating character string type.
Step 230: by the second of a plurality of corresponding log information template of log information and a plurality of log information
Part is stored respectively to database.
Comparison can will distinguished to multiple log informations in the embodiment of the present application, identify multiple log informations
First part and second part.The corresponding log information template of multiple log informations is stored in distributed data base 112
An entity in, and the part changed in each log information is stored in another entity of distributed data base 112.
In above-mentioned technical proposal, original log information can be handled, be changed in identifying multiple log informations
Part and constant part after, part constant in each log information is only stored one, and by each log information
It is condensed into part and log information template identification, the relevant label information of the log information etc. including its variation.To make
Log information after must simplifying does not have to carry a lot of character string in template, and it is empty to reduce storage shared by log information
Between, save the storage resource consumption in log service system.
Optionally, in some embodiments, template identification corresponding to each log information template can be generated, and will
Each log information simplifies to include: log information template identification, variate-value, timestamp, label information etc..In order to user
It, can be by day for including in each log information during being retrieved to log information and being reduced into raw information
Will information model mark, gets log information template.So as to include according in log information template and log information
Timestamp, variate-value log information is reverted to original state.
It should be understood that the timestamp for including in log information after simplifying can be what log information itself recorded.Label
Information can be the log collection being also possible at least one node by log information itself note and act on behalf of according to log information
What the node at place and the attribute of the application service operated above were got, it specifically please refers to and above label information is retouched
It states, details are not described herein again.
The embodiment of the present application can extract the feature of an information by information fingerprint extraction algorithm, and according to this information
Feature convert thereof into one group of code.The code can be used as unique template mark corresponding to each log information module
Know, identical log information template mark having the same under different nodes can be made in this way.As an example rather than
It limits, can be generated corresponding to each log information module only by summary info (message-digest, MD5) algorithm
One template identification.
It should be understood that MD5 algorithm is a kind of Cryptographic Hash Function being widely used, it can produce out 128 (16 words
Section) hashed value (hash value), for ensuring that information transmission is complete consistent.Below in conjunction with specific embodiment to passing through
The realization process that MD5 algorithm generates unique template identification corresponding to each log information module is described in detail,
Details are not described herein again.
In the embodiment of the present application, generated by using information fingerprint technique unique corresponding to each log information template
Template identification, identical log information template mark having the same under different nodes can be made, so as to reduce
Be unified for overhead brought by the communication that each log information template is distributed between the cross-node of its corresponding mark and
Complexity.
It is specific to being simplified described in the above process to log information below with reference to specific log information content
Realization process is described in detail.For ease of description, below to extract a log information from 3 original log information
It is illustrated for template.
It should be understood that following example is used for the purpose of helping skilled in the art to understand the embodiment of the present application, and have to
By the embodiment of the present application be limited to illustrated by specific value or concrete scene.Those skilled in the art are according to below given
Example, it is clear that the modification or variation of various equivalences can be carried out, such modification or variation also fall into the model of the embodiment of the present application
In enclosing.
Original log information 1:[2018-11-09 13:39:14.696] [10.0.26.102] [INF0] [bulk_
thread-2][BulkHandlerRunable:submits 138]get data from queue timeout！
Original log information 2:[2018-11-09 13:39:15.698] [10.0.26.102] [INF0] [bulk_
thread-5][BulkHandlerRunable:submits 138]get data from queue timeout！
Original log information 3:[2018-11-09 13:39:16.066] [10.0.26.102] [INF0] [bulk_
thread-3][BulkHandlerRunable:submits 138]get data from queue timeout！
Step 1: identifying the part and constant portion changed in each original log information in 3 original log information
Point.
It can be identified by schema extraction algorithm in the embodiment of the present application and extract each original in 3 original log information
The part and constant part changed in beginning log information.It is non-limiting as example, iteration subregion Web log mining can be passed through
(iterative partitioning log mining, IPLOM) algorithm is to identifying each original in 3 original log information
The part and constant part changed in beginning log information, and can be replaced with occupy-place identifier every in 3 original log information
The part of a original log information change.
Schema extraction algorithm by step 1, the recognition result after comparing to above-mentioned 3 original log information is such as
Shown in lower:
Original log information 1 is as follows relative to the comparison result of original log information 2,3:
Constant part: [10.0.26.102] [INF0] [bulk_thread-] [BulkHandlerRunable:
submits 138]get data from queue timeout！
The part of variation: 2, [2018-11-09 13:39:14.696]
With the numerical part changed in occupy-place identifier %d replacement original log information 1, original log is indicated with tm field
The field of logging time in information 1, log original log information 1 can indicate are as follows:
Tm [10.0.26.102] [INF0] [bulk_thread-%d] [BulkHandler Runable:submits
138]get data from queue timeout！
Wherein, %d=2, tm=[2018-11-09 13:39:14.696]
Original log information 2 is as follows relative to the comparison result of original log information 1,3:
Constant part: [10.0.26.102] [INF0] [bulk_thread-] [BulkHandlerRunable:
submits 138]get data from queue timeout！
The part of variation: 5, [2018-11-09 13:39:15.698]
With the part changed in occupy-place identifier %d replacement original log information 2, original log information is indicated with tm field
The field of logging time in 2, original log information 2 can indicate are as follows:
Tm [10.0.26.102] [INF0] [bulk_thread-%d] [BulkHandler Runable:submits
138]get data from queue timeout！
Wherein, %d=5, tm=[2018-11-09 13:39:15.698].
Original log information 3 is as follows relative to the comparison result of original log information 1,2:
Constant part: [10.0.26.102] [INF0] [bulk_thread-] [BulkHandlerRunable:
submits 138]get data from queue timeout！
The part of variation: 3, [2018-11-09 13:39:16.066]
With the part changed in occupy-place identifier %d replacement original log information 3, original log information is indicated with tm field
The field of logging time in 3, original log information 3 can indicate are as follows:
Tm [10.0.26.102] [INF0] [bulk_thread-%d] [BulkHandler Runable:submits
138]get data from queue timeout！
Wherein, %d=3, tm=[2018-11-09 13:39:16.066].
In conclusion the log information template identified in 3 original log information are as follows: tm [INF0] [bulk_
Thread-%d] [BulkHandlerRunable:submits 138] get data from queue timeout！
Step 2: generating the corresponding unique mark of log information template.
Under the distributed scene including at least one node, there may be identical log information moulds for different nodes
Plate, so that the mark of log information template and the template corresponds.Digital Fingerprinting Technology can be used in the embodiment of the present application
Realize that the mark of log information template and the template corresponds, for example, it is corresponding to generate log information template using MD5 algorithm
Unique mark.
Specifically, the character string of the above-mentioned log information template of MD5 algorithm is calculated, generation 128 (16 byte) dissipates
Train value, the hashed value can be used as the corresponding unique mark of above-mentioned log information template.
It can also be selected from the hashed value for generating one 128 (16 byte) by MD5 algorithm in the embodiment of the present application
64 out (8 byte) hashed values, and using 64 hashed values as corresponding unique mark of actual log information template.
There are many implementations that 64 hashed values are selected from 128 hashed values, as an example, selects from 128 hashed values
64 odd number bit positions are selected out as 64 hashed values.As another example, 64 idols are selected from 128 hashed values
The part of numerical digit is as 64 hashed values.As another example, intermediate 64 are selected from 128 hashed values as 64
Position hashed value.As another example, it is added after 128 hashed values can also being folded and obtains 64 hashed values.
For example, can indicate the mark of above-mentioned log information template with tid field.
{ tid=5085657271133051000
Tm [INF0] [bulk_thread-%d] [BulkHandlerRunable:submits 138] get data
from queue timeout！}
Step 3: the field of logging time in conversion original log information.
It should be understood that step 3 is optional.
In order to reduce the occupied storage resource of character string in the field for indicating logging time, original log can be believed
Logging time in breath is converted, and logging time is converted into the millisecond offset numbers of fiducial time.It as an example, can be with
Using January 1 as fiducial time in 1970, the time recorded in each primitive beginning log information was calculated relative to fiducial time
Number of seconds deviant.
The tm=for example, tm=1541770754696 in calculated original log information 1, in original log information 2
1541770755698, the tm=1541770756066 in original log information 3.
Step 4: indicating the part of the variation in each original log information.
The field for storing variate-value in each original log information can be increased in the embodiment of the present application, shown as one
Example, indicates variate-value list character string with mg field.
Variate-value for example, the variable value list character string mg=(2) in original log information 1, in original log information 2
List character string mg=(5), the variable value list character string mg=(3) in original log information 3.
The field for storing the number of variate-value in each original log information can also be increased in the embodiment of the present application, made
For an example, the number of variate-value is indicated with VL field.
For example, the number VL=1 of the variate-value in original log information 1, the number of the variate-value in original log information 2
VL=1, the number VL=1 of the variate-value in original log information 3.
Step 5: adding the label information of each original log information.
Relevant label information can be added to each original log information in the embodiment of the present application, in order to by each
Kind label information gets the log information of needs.
For ease of description, the embodiment of the present application is using label information as the path of the journal file where log information, day
It is illustrated for the IP address of node where will information.
IP=by taking label information is the IP address of log information place node as an example, in original log information 1
10.0.26.102, the IP=10.0.26.102 in original log information 2, the IP=in original log information 3
10.0.26.102。
By taking the path of the journal file where label information is log information as an example, the embodiment of the present application can be to each
The path of journal file carries out information fingerprint extraction, forms the corresponding unique mark in path of each journal file, from
And storage resource can be saved.Specific information fingerprint extracts and generates the process of mark and the calculating of log information template identification
Process is similar, specifically please refers to the generating process of log information template identification, details are not described herein again.
As an example, the path of the journal file where indicating log information with pid field.For example, original log
Pid=864995200973638000 in information 1, the pid=864995200973638000 in original log information 2, it is former
Pid=864995200973638000 in beginning log information 3.
By above-mentioned steps 1- step 5, by above-mentioned 3 original log information reductions at: timestamp+template identification+variable+
Associated tag information.And template identification and template can be respectively stored in the different tables of distributed data base 112.
As an example, original log information can be split into 3 parts to store respectively.For example, table 1 is for storing
Log information after simplifying, table 2 are used to store the path of the journal file where log information, and table 3 is original for storing 3
The template of log information.
The content stored in table 1 is as follows:
Log information 1 after simplifying can indicate are as follows:
Log information 2 after simplifying can indicate are as follows:
Log information 3 after simplifying can indicate are as follows:
The content stored in table 2 is as follows:
The content stored in table 3 is as follows:
It should be understood that in the embodiment of the present application, step some or all of can be executed in the embodiment of the present application, these
Step or operation are only examples, and the deformation of other operations or various operations can also be performed in the embodiment of the present application.In addition, each
Step can be executed according to the different sequences that the embodiment of the present application is presented, and it is possible to not really want to execute the application implementation
All operationss in example.
Optionally, in some embodiments, user is passing through 111 pairs of log services API storages in the embodiment of the present application
During log information is retrieved, the search of keyword can be carried out, in log information template so as to quickly search
The log information needed to user, reduces the time of search.
Optionally, in some embodiments, the embodiment of the present application can also be at 111 end log services API to the essence of storage
Log information after letter is restored, and arrives original log information in order to which user is available, and can be by original
Log information analysis, understand node in operation often occur or the accidental problem.
Specifically, as shown in Figure 1, user can be defeated by 111 end log services API in distributed data base 112
Enter various label informations and log keyword to get the log information of needs.Distributed data base 112 can be according to pass
The log information template identification tid for including in key word obtains associated log information template and variable value list character string
Mg, and according to the log information template and variable value list character string mg got, variable value list character string mg is automatic
The position of the occupy-place identifier of log information template is brought into, to restore original log information.
It should be noted that the distributed data base 112 in the embodiment of the present application can be distributed relational data
Library is also possible to the non-relational database with join function.Distributed relevant database or with join function
Non-relational database associated log information template can be obtained by log information template identification tid and variate-value arranges
Table character string mg.
Optionally, in some embodiments, user can service the interface that API 111 is provided by browser usage log
It is interacted with log service system 100.
Specifically, user can be requested by browser input inquiry, it may include for needing to look into the inquiry request
The keyword of the log information of inquiry and/or relevant label information.Database can be according to the log information in inquiry request
Keyword and/or relevant label information obtain the log information of storage from the table of database, and can be by the log of storage
Information feeds back to user.User not only can by the analysis to log information, understand node in operation often occur or
The accidental problem can also excavate potential system by the regularity of distribution of mark corresponding to analysis log information template
System is abnormal, and issues warning reminding in advance.
It should be understood that the keyword for including in inquiry request can be some character string in log information.
As an example, the inquiry request of user's input is to inquire the log information in " on January 1st, 2018 ", database
The value of tm can be determined according to the time " on January 1st, 2018 " of inquiry after receiving inquiry request.And it can be with
Satisfactory log information is obtained from the log information stored in the table 1 of database according to the value of tm.Meanwhile database
Join function log information template corresponding to the log information template identification in the log information can be obtained from table 3,
The log information stored in table 1 is brought into log information template, generates original log information, and original log can be believed
Breath feeds back to user.As another example, user can also a word in inquiry request in input journal information model
Symbol string, database can obtain log information template corresponding to the log information template identification in the log information in table 3.
The log with the log information template identification is obtained in the log information that the join function of database can also be stored from table 1
The log information stored in table 1 is brought into log information template by information, generates original log information, and can will be original
Log information feeds back to user.
It is below that the log information for inquiring IP address to generate on " 10.0.26.102 " is made with the inquiry request that user inputs
For example, detailed explanation is carried out to the process of user query log information
It include: IP=10.0.26.102 in the inquiry request that user is inputted by log services API 111.Database 112
According to IP=10.0.26.102, the log letter of IP=10.0.26.102 can be got from the log information stored in table 1
Breath, as follows:
The join function of database 112 can also be tid=according in above-mentioned log information
5085657271133051000, the corresponding log information template of tid=5085657271133051000 is obtained from table 3, such as
Shown in lower:
Database 112 can restore the timestamp information of original carrying in log information according to the tm in log information.
For example, tm=1541770754696 is reduced into [2018-11-09 13:39:14.696], also by tm=1541770755698
Original be reduced at [2018-11-09 13:39:15.698], by tm=1541770756066 [2018-11-09 13:39:
16.066]。
Database 112 can also by the timestamp information of original carrying and it is above-mentioned got from table 1 meet IP=
10.0.26.102 log information is brought into log information template, forms original log information.Original log information is provided
To the front-end interface of log services API 111, to realize the reduction to original log information.
The original log information for being supplied to the front-end interface of log services API 111 is as follows:
Log information 1:
[2018-11-09 13:39:14.696][10.0.26.102][INF0][bulk_thread-2]
[BulkHandlerRunable:submits 138]get data from queue timeout！
Log information 2:
[2018-11-09 13:39:15.698][10.0.26.102][INF0][bulk_thread-5]
[BulkHandlerRunable:submits 138]get data from queue timeout！
Log information 3:
[2018-11-09 13:39:16.066][10.0.26.102][INF0][bulk_thread-3]
[BulkHandlerRunable:submits 138]get data from queue timeout！
Above in association with Fig. 1 to Fig. 2, the embodiment of the present application is described in detail, the method and inquiry of storage log information are provided
The Installation practice of the application is described in detail below in conjunction with Fig. 3-Fig. 4 in the method for log information.It should be understood that embodiment of the method
Description corresponded to each other with the description of Installation practice, therefore, the part being not described in detail may refer to previous methods embodiment.
Fig. 3 is a kind of structural schematic figure of device 300 for storing log information provided by the embodiments of the present application.The dress
Setting 300 includes: identification module 310, processing module 320, memory module 330.
Identification module 310, for identifying the first part and of a plurality of log information from a plurality of log information
Two parts, wherein the first part of a plurality of log information is to be less than or equal to the first threshold in a plurality of log information
It is worth identical part in a log information, the second part of a plurality of log information is to be less than in a plurality of log information
Or it is equal to different part in second threshold log information；
Processing module 320, for replacing the second part of a plurality of log information by occupy-place identifier, described in formation
The corresponding log information template of a plurality of log information；
Memory module 330, for believing a plurality of corresponding log information template of log information and a plurality of log
The second part of breath is stored respectively to database.
Optionally, in some embodiments, described device 300 further include:
Generation module 340, for generating the mark of the corresponding log information template of a plurality of log information.
Optionally, in some embodiments, the memory module 330 is specifically used for: by the of a plurality of log information
The mark of two parts and the log information template is stored in the first space of the database；By the log information template and
The mark of the log information template is stored in the second space of the database.
It should be understood that the device 300 of the storage log information of the embodiment of the present application can pass through specific integrated circuit
(application-specific integrated circuit, ASIC) is realized or programmable logic device
(programmable logic device, PLD) is realized, above-mentioned PLD can be complicated process logical device (complex
Programmable logical device, CPLD), field programmable gate array (field-programmable gate
Array, FPGA), Universal Array Logic (generic array logic, GAL) or any combination thereof.Software can also be passed through
When realizing the method for storage log information shown in Fig. 2, the device 300 and its modules for storing log information may be soft
Part module.
Fig. 4 is a kind of structural schematic figure of device 400 for storing log information provided by the embodiments of the present application.Storage
The device 400 of log information includes at least one calculate node 410, and calculate node 410 may include processing unit 411 and communication
Interface 412, processing unit 411 is for executing function defined in various software programs, for example, for realizing storage log information
Function.Communication interface 412 is used to carry out communication interaction with other calculate nodes, and other equipment can be other physical services
Device, specifically, communication interface 412 can be adapter.
Optionally, which can also include input/output interface 413, and input/output interface 413 is connected with
Input-output apparatus, information for receiving input export operating result.Input/output interface 413 can be mouse, key
Disk, display or CD-ROM drive etc..Optionally, which can also include additional storage 414, also commonly referred to as outer
It deposits, the storage medium of additional storage 414 can be magnetic medium (for example, floppy disk, hard disk, tape), optical medium (such as light
Disk) or semiconductor medium (such as solid state hard disk) etc..
Optionally, calculate node 410 can also include bus 415.Wherein, processing unit 411, communication interface 412, defeated
Enter/output interface 413, additional storage 414 can connect by bus 415.Bus 415 can be Peripheral Component Interconnect standard
(peripheral component interconnect, PCI) bus or expanding the industrial standard structure (extended
Industry standard architecture, EISA) bus etc..Bus 415 can be divided into address bus, data/address bus,
Control bus etc..Only to be indicated with a line in Fig. 4, it is not intended that an only bus or a type of convenient for indicating
Bus.
Processing unit 411 can there are many specific implementation forms, such as processing unit 411 may include 4112 He of processor
Memory 4111, processor 4112 execute the correlation of embodiment shown in Fig. 2 according to the program instruction stored in memory 4111
Operation.Processor 4112 can be central processing unit (central processing unit, CPU).The processor can be with
It is other general processors, digital signal processor (digital signal processor, DSP), specific integrated circuit
(application specific integrated circuit, ASIC), ready-made programmable gate array (field
Programmable gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.Or the processor 410 uses one or more integrated circuits, for executing relative program, to realize the embodiment of the present application
Provided technical solution.
It should be understood that the processor 4112 of calculate node 410 can by the program instruction that is stored in memory 4111 come
Run at least one of identification module 310 as shown in Figure 3, processing module 320, memory module 330.
It should also be understood that the memory 4111 or additional storage 414 of calculate node 410 can also be stored with described in Fig. 2
Database.
Above and other operation and/or function of each unit in the device 400 of log information is stored respectively for reality
The corresponding process of method in existing Fig. 2, for sake of simplicity, details are not described herein.
Fig. 5 is a kind of structural schematic figure of the device 500 of inquiry log information provided by the embodiments of the present application.This is looked into
Asking the 500 of log information includes: receiving module 510, acquisition module 520, return module 530.
Receiving module 510, for receiving inquiry request, the inquiry request is used to inquire the log letter stored in database
Breath；
Module 520 is obtained, for obtaining a plurality of log of storage from the database according to the inquiry request
The second part of information and the corresponding log information template of a plurality of log information；By second of a plurality of log information
Point bringing the corresponding log information template of a plurality of log information into obtains the corresponding log information of the inquiry request；
It optionally, can also include return module 530, for returning to the correspondence to the client for issuing the inquiry request
Log information.
Return module 530 and receiving module 510 can also be realized by same module.
Optionally, in some embodiments, described device 500 further include:
Identification module 540, for identifying the first part of a plurality of log information from a plurality of log information
With the second part；
Processing module 550 is formed described more for replacing the second part of a plurality of log information with occupy-place identifier
The corresponding log information template of log information；
Memory module 560, for believing a plurality of corresponding log information template of log information and a plurality of log
The second part of breath is stored respectively to database.
Optionally, in some embodiments, described device 500 further include:
Generation module 570, for generating the mark of the corresponding log information template of a plurality of log information.
Optionally, in some embodiments, memory module 560 is specifically used for: by second of a plurality of log information
Divide the first space that the database is stored in the mark of the log information template；By the log information template and described
The mark of log information template is stored in the second space of the database.
It should be understood that the device 500 of the inquiry log information of the embodiment of the present application can pass through specific integrated circuit
(application-specific integrated circuit, ASIC) is realized or programmable logic device
(programmable logic device, PLD) is realized, above-mentioned PLD can be complicated process logical device (complex
Programmable logical device, CPLD), field programmable gate array (field-programmable gate
Array, FPGA), Universal Array Logic (generic array logic, GAL) or any combination thereof.Software can also be passed through
When realizing the method for inquiry log information, the device 500 and its modules of inquiry log information may be software module.
Fig. 6 is a kind of structural schematic figure of the device 600 of inquiry log information provided by the embodiments of the present application.Inquiry
The device 600 of log information includes at least one calculate node 610, and calculate node 610 may include processing unit 611 and communication
Interface 612, processing unit 611 is for executing function defined in various software programs, for example, for realizing storage log information
Function.Communication interface 412 is used to carry out communication interaction with other calculate nodes, and other equipment can be other physical services
Device, specifically, communication interface 612 can be adapter.
Optionally, which can also include input/output interface 613, and input/output interface 413 is connected with
Input-output apparatus, information for receiving input export operating result.Input/output interface 613 can be mouse, key
Disk, display or CD-ROM drive etc..Optionally, which can also include additional storage 614, also commonly referred to as outer
It deposits, the storage medium of additional storage 614 can be magnetic medium (for example, floppy disk, hard disk, tape), optical medium (such as light
Disk) or semiconductor medium (such as solid state hard disk) etc..
Optionally, calculate node 610 can also include bus 615.Wherein, processing unit 611, communication interface 612, defeated
Enter/output interface 613, additional storage 614 can connect by bus 615.The function and realization of bus 615 and bus 415
Mode is similar.
Processing unit 611 can there are many specific implementation forms, such as processing unit 611 may include 6112 He of processor
Memory 6111, processor 6112 execute the relevant operation of above-described embodiment according to the program instruction stored in memory 6111.
Processor 6112 can be central processing unit (central processing unit, CPU).The processor can also be it
Its general processor, digital signal processor (digital signal processor, DSP), specific integrated circuit
(application specific integrated circuit, ASIC), ready-made programmable gate array (field
Programmable gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.Or the processor 610 uses one or more integrated circuits, for executing relative program, to realize the embodiment of the present application
Provided technical solution.
It should be understood that the processor 6112 of calculate node 610 can by the program instruction that is stored in memory 6111 come
At least one of run receiving module 510 as shown in Figure 5, obtain module 520, return module 530.
It should also be understood that the memory 6111 or additional storage 614 of calculate node 610 can also be stored with described in Fig. 2
Database.
Above and other operation and/or function of each unit in the device 600 of inquiry log information is respectively for reality
The corresponding process of the method for existing above-mentioned inquiry log information, for sake of simplicity, details are not described herein.
Above-described embodiment can be realized wholly or partly by software, hardware, firmware or any other combination.When
When using software realization, above-described embodiment can be realized entirely or partly in the form of a computer program product.The calculating
Machine program product includes one or more computer instructions.When loading on computers or executing the computer program instructions,
It entirely or partly generates according to process or function described in the embodiment of the present application.The computer can for general purpose computer,
Special purpose computer, computer network or other programmable devices.The computer instruction can store to be deposited computer-readable
In storage media, or from a computer readable storage medium to the transmission of another computer readable storage medium, for example, described
Computer instruction can pass through wired (such as coaxial cable, light from a web-site, computer, server or data center
Fine, Digital Subscriber Line (DSL)) or wireless (such as infrared, wireless, microwave etc.) mode to another web-site, computer, clothes
Business device or data center are transmitted.The computer readable storage medium can be any available Jie that computer can access
Matter includes either the data storage devices such as server, the data center of one or more usable medium set.Described use is situated between
Matter can be magnetic medium (for example, floppy disk, hard disk, tape), optical medium (for example, DVD) or semiconductor medium.Semiconductor
Medium can be solid state hard disk (solid state drive, SSD).
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure
Member and algorithm steps can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually
It is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technician
Each specific application can be used different methods to achieve the described function.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed systems, devices and methods, it can be with
It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the unit
It divides, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components
It can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, it is shown or
The mutual coupling, direct-coupling or communication connection discussed can be through some interfaces, the indirect coupling of device or unit
It closes or communicates to connect, can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
Claims (18)
1. a kind of method for storing log information, which is characterized in that the described method includes:
First part and the second part of a plurality of log information are identified from a plurality of log information, wherein described a plurality of
The first part of log information is identical less than or equal in first threshold log information in a plurality of log information
Part, the second part of a plurality of log information are to be less than or equal to second threshold log in a plurality of log information
Different part in information；
The second part of a plurality of log information is replaced with occupy-place identifier, forms the corresponding log of a plurality of log information
Information model, the log information template include the first part of a plurality of log information；
The second part of a plurality of corresponding log information template of log information and a plurality of log information is stored respectively
To database.
2. the method according to claim 1, wherein the method also includes:
Generate the mark of the corresponding log information template of a plurality of log information.
3. according to the method described in claim 2, it is characterized in that, described by the corresponding log information of a plurality of log information
The second part of template and a plurality of log information is stored respectively to database, comprising:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database
One space；
The mark of the log information template and the log information template is stored in the second space of the database.
4. a kind of method of inquiry log information, which is characterized in that the method is applied to log information inquiry system, the day
Will information query system includes database, the database purchase have a plurality of log information second part and a plurality of log
The corresponding log information template of information, the log information template includes the first part of a plurality of log information, described more
The first part of log information be less than or equal in a plurality of log information it is identical in first threshold log information
Part, the second part of a plurality of log information is to be less than or equal to second threshold item day in a plurality of log information
Different part in will information, which comprises
Inquiry request is received, the inquiry request is for inquiring the log information stored in database；
The second part of a plurality of log information of storage and described is obtained from the database according to the inquiry request
The corresponding log information template of a plurality of log information；
Bring the second part of a plurality of log information into a plurality of log information corresponding log information template acquisition pair
The log information answered.
5. according to the method described in claim 4, it is characterized in that, the method is also wrapped before the reception inquiry request
It includes:
Identified from a plurality of log information a plurality of log information the first part and a plurality of log information
The second part；
The second part of a plurality of log information is replaced with occupy-place identifier, and it is corresponding to form a plurality of log information
Log information template；
The second part of a plurality of corresponding log information template of log information and a plurality of log information is stored respectively
To the database.
6. according to the method described in claim 5, it is characterized in that, the method also includes:
Generate the mark of the corresponding log information template of a plurality of log information.
7. according to the method described in claim 6, it is characterized in that, the method also includes:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database
One space；
The mark of the log information template and the log information template is stored in the second space of the database.
8. a kind of device for storing log information, which is characterized in that described device includes:
Identification module, for identifying first part and the second part of a plurality of log information from a plurality of log information,
Wherein, the first part of a plurality of log information is to be less than or equal to first threshold log in a plurality of log information
Identical part in information, the second part of a plurality of log information are to be less than or equal to the in a plurality of log information
Different part in two threshold value log informations；
Processing module forms a plurality of day for replacing the second part of a plurality of log information by occupy-place identifier
The corresponding log information template of will information, the log information template include the first part of a plurality of log information；
Memory module, for by the second of a plurality of corresponding log information template of log information and a plurality of log information
Part is stored respectively to database.
9. device according to claim 8, which is characterized in that described device further include:
Generation module, for generating the mark of the corresponding log information template of a plurality of log information.
10. device according to claim 9, which is characterized in that the memory module is specifically used for:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database
One space；
The mark of the log information template and the log information template is stored in the second space of the database.
11. a kind of device of inquiry log information, which is characterized in that described device is applied to log information inquiry system, described
Log information inquiry system includes database, the database purchase have a plurality of log information second part and a plurality of day
The corresponding log information template of will information, the log information template includes the first part of a plurality of log information, described
The first part of a plurality of log information is to be less than or equal to phase in first threshold log information in a plurality of log information
Same part, the second part of a plurality of log information are to be less than or equal to second threshold item in a plurality of log information
Different part, described device include: in log information
Receiving module, for receiving inquiry request, the inquiry request is for inquiring the log information stored in database；
Module is obtained, the of a plurality of log information for obtaining storage from the database according to the inquiry request
Two parts and the corresponding log information template of a plurality of log information；Bring the second part of a plurality of log information into institute
It states the corresponding log information template of a plurality of log information and obtains corresponding log information.
12. device according to claim 11, which is characterized in that described device further include:
Identification module, for identifying the first part and described of a plurality of log information from a plurality of log information
Two parts；
Processing module forms a plurality of log for replacing the second part of a plurality of log information with occupy-place identifier
The corresponding log information template of information；
Memory module, for by the second of a plurality of corresponding log information template of log information and a plurality of log information
Part is stored respectively to database.
13. device according to claim 12, which is characterized in that described device further include:
Generation module, for generating the mark of the corresponding log information template of a plurality of log information.
14. device according to claim 13, which is characterized in that memory module is specifically used for:
The mark of the second part of a plurality of log information and the log information template is stored in the of the database
One space；
The mark of the log information template and the log information template is stored in the second space of the database.
15. a kind of device for storing log information, which is characterized in that including at least one calculate node, each calculate node packet
Processor and memory are included, when the device of the storage log information is run, the processor runs the meter in the memory
Calculation machine is executed instruction in the method as described in any one of claims 1 to 3 of execution.
16. a kind of device of inquiry log information, which is characterized in that including at least one calculate node, each calculate node packet
Processor and memory are included, when the device of the storage log information is run, the processor runs the meter in the memory
Calculation machine is executed instruction in the method as described in any one of claim 4 to 7 of execution.
17. a kind of non-volatile computer readable storage medium storing program for executing that computer is non-transient, which is characterized in that including computer journey
Sequence, when the computer program is run at least one calculate node, so that at least one calculate node executes such as right
It is required that method described in any one of 1 to 3.
18. a kind of non-volatile computer readable storage medium storing program for executing that computer is non-transient, which is characterized in that including computer journey
Sequence, when the computer program is run at least one calculate node, so that at least one calculate node executes such as right
It is required that method described in any one of 4 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910106607.0A CN109885545A (en) | 2019-02-02 | 2019-02-02 | It stores, the method, apparatus of inquiry log information |
| PCT/CN2019/107127 WO2020155651A1 (en) | 2019-02-02 | 2019-09-21 | Method and device for storing and querying log information |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910106607.0A CN109885545A (en) | 2019-02-02 | 2019-02-02 | It stores, the method, apparatus of inquiry log information |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109885545A true CN109885545A (en) | 2019-06-14 |
Family
ID=66927813
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910106607.0A Pending CN109885545A (en) | 2019-02-02 | 2019-02-02 | It stores, the method, apparatus of inquiry log information |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109885545A (en) |
| WO (1) | WO2020155651A1 (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020155651A1 (en) * | 2019-02-02 | 2020-08-06 | 华为技术有限公司 | Method and device for storing and querying log information |
| CN112152823A (en) * | 2019-06-26 | 2020-12-29 | 北京易真学思教育科技有限公司 | Website operation error monitoring method and device and computer storage medium |
| CN113420003A (en) * | 2021-06-30 | 2021-09-21 | 中国航空油料有限责任公司 | Method, device, equipment and medium for processing data interaction log |
| CN113472555A (en) * | 2020-03-30 | 2021-10-01 | 华为技术有限公司 | Fault detection method, system, device, server and storage medium |
| CN116894021A (en) * | 2023-05-24 | 2023-10-17 | 北京优特捷信息技术有限公司 | Log data storage method, query method, device, equipment and medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116644039B (en) * | 2023-05-25 | 2023-12-19 | 安徽继远软件有限公司 | Automatic acquisition and analysis method for online capacity operation log based on big data |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080072140A1 (en) * | 2006-07-05 | 2008-03-20 | Vydiswaran V G V | Techniques for inducing high quality structural templates for electronic documents |
| US20110219046A1 (en) * | 2010-03-05 | 2011-09-08 | Igor Nesmyanovich | System, method and computer program product for managing data storage and rule-driven communications for a plurality of tenants |
| CN103544176A (en) * | 2012-07-13 | 2014-01-29 | 百度在线网络技术（北京）有限公司 | Method and device for generating page structure template corresponding to multiple pages |
| CN104331487A (en) * | 2014-11-13 | 2015-02-04 | 携程计算机技术（上海）有限公司 | Method and device for processing logs |
| CN108241658A (en) * | 2016-12-24 | 2018-07-03 | 北京亿阳信通科技有限公司 | A kind of logging mode finds method and system |
| CN108694206A (en) * | 2017-04-11 | 2018-10-23 | 富士通株式会社 | Information processing method and device |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8533193B2 (en) * | 2010-11-17 | 2013-09-10 | Hewlett-Packard Development Company, L.P. | Managing log entries |
| US8620928B1 (en) * | 2012-07-16 | 2013-12-31 | International Business Machines Corporation | Automatically generating a log parser given a sample log |
| US9166866B2 (en) * | 2013-04-30 | 2015-10-20 | Microsoft Technology Licensing, Llc | Hydration and dehydration with placeholders |
| CN109885545A (en) * | 2019-02-02 | 2019-06-14 | 华为技术有限公司 | It stores, the method, apparatus of inquiry log information |
-
2019
- 2019-02-02 CN CN201910106607.0A patent/CN109885545A/en active Pending
- 2019-09-21 WO PCT/CN2019/107127 patent/WO2020155651A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080072140A1 (en) * | 2006-07-05 | 2008-03-20 | Vydiswaran V G V | Techniques for inducing high quality structural templates for electronic documents |
| US20110219046A1 (en) * | 2010-03-05 | 2011-09-08 | Igor Nesmyanovich | System, method and computer program product for managing data storage and rule-driven communications for a plurality of tenants |
| CN103544176A (en) * | 2012-07-13 | 2014-01-29 | 百度在线网络技术（北京）有限公司 | Method and device for generating page structure template corresponding to multiple pages |
| CN104331487A (en) * | 2014-11-13 | 2015-02-04 | 携程计算机技术（上海）有限公司 | Method and device for processing logs |
| CN108241658A (en) * | 2016-12-24 | 2018-07-03 | 北京亿阳信通科技有限公司 | A kind of logging mode finds method and system |
| CN108694206A (en) * | 2017-04-11 | 2018-10-23 | 富士通株式会社 | Information processing method and device |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020155651A1 (en) * | 2019-02-02 | 2020-08-06 | 华为技术有限公司 | Method and device for storing and querying log information |
| CN112152823A (en) * | 2019-06-26 | 2020-12-29 | 北京易真学思教育科技有限公司 | Website operation error monitoring method and device and computer storage medium |
| CN113472555A (en) * | 2020-03-30 | 2021-10-01 | 华为技术有限公司 | Fault detection method, system, device, server and storage medium |
| CN113472555B (en) * | 2020-03-30 | 2022-09-23 | 华为技术有限公司 | Fault detection method, system, device, server and storage medium |
| CN113420003A (en) * | 2021-06-30 | 2021-09-21 | 中国航空油料有限责任公司 | Method, device, equipment and medium for processing data interaction log |
| CN116894021A (en) * | 2023-05-24 | 2023-10-17 | 北京优特捷信息技术有限公司 | Log data storage method, query method, device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2020155651A1 (en) | 2020-08-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109885545A (en) | It stores, the method, apparatus of inquiry log information | |
| US11860874B2 (en) | Multi-partitioning data for combination operations | |
| US11636105B2 (en) | Generating a subquery for an external data system using a configuration file | |
| US11586692B2 (en) | Streaming data processing | |
| US11321321B2 (en) | Record expansion and reduction based on a processing task in a data intake and query system | |
| US11663227B2 (en) | Generating a subquery for a distinct data intake and query system | |
| US20200364223A1 (en) | Search time estimate in a data intake and query system | |
| US11615104B2 (en) | Subquery generation based on a data ingest estimate of an external data system | |
| US11604795B2 (en) | Distributing partial results from an external data system between worker nodes | |
| US10447772B2 (en) | Managed function execution for processing data streams in real time | |
| US11151137B2 (en) | Multi-partition operation in combination operations | |
| US11163758B2 (en) | External dataset capability compensation | |
| US11461334B2 (en) | Data conditioning for dataset destination | |
| US11232100B2 (en) | Resource allocation for multiple datasets | |
| US11394767B2 (en) | Central repository of configuration files and two-way replication of search node configuration files | |
| US20190138638A1 (en) | Task distribution in an execution node of a distributed execution environment | |
| EP2182448A1 (en) | Federated configuration data management | |
| US11829381B2 (en) | Data source metric visualizations | |
| WO2020087082A1 (en) | Trace and span sampling and analysis for instrumented software | |
| JP2013140510A (en) | Data processing method, decentralized processing system, and program | |
| US11681707B1 (en) | Analytics query response transmission | |
| US20230018723A1 (en) | Cascading payload replication | |
| Theeten et al. | Chive: Bandwidth optimized continuous querying in distributed clouds | |
| US11604789B1 (en) | Bi-directional query updates in a user interface | |
| US10324818B2 (en) | Data analytics correlation for heterogeneous monitoring systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190614 |
|
| RJ01 | Rejection of invention patent application after publication |