CN101339531A - Method and device for processing log file - Google Patents
Method and device for processing log file Download PDFInfo
- Publication number
- CN101339531A CN101339531A CNA2008101182868A CN200810118286A CN101339531A CN 101339531 A CN101339531 A CN 101339531A CN A2008101182868 A CNA2008101182868 A CN A2008101182868A CN 200810118286 A CN200810118286 A CN 200810118286A CN 101339531 A CN101339531 A CN 101339531A
- Authority
- CN
- China
- Prior art keywords
- log information
- correlation
- journal file
- degree
- incident
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Abstract
The invention discloses a method for disposing log files and a device thereof. The invention has the technical proposal that events in the log information are analyzed by taking the log files that are specified to be analyzed as a basis and an event model database is built by taking the log information that belongs to the same event as a model; and the log files that are specified to be disposed are analyzed and consolidated according to the built event model database so as to obtain the log files which are recorded by taking the event as a whole. The method for disposing the log files and the device provided by the invention are characterized in that the model that denotes the events can be obtained by analyzing the log files, the log files that are specified to be disposed are disposed according the model, the log information in the log files is consolidated so as to ensure that the log information is recorded by taking the log information that denotes the same event as a whole, thereby leading the disposed log files to clearly and perspicuously reflect the events that happens in a system or in a software system and being convenient for the well-off going of the log analyzing work.
Description
Technical field
The present invention relates to the log analysis technology, refer to a kind of method and device of handling journal file especially.
Background technology
Be the epoch of computer industry develop rapidly now, various softwares or system are tinkling of jades meets the eye on every side, but it is that some daily records all can be printed by each ripe software or system that individual common characteristic is arranged.When the operation of software or system went wrong, it is most important that these daily records will become, and many times is the necessary tool of positioning software or system mistake.
Generally, each log information that the same incident of continuous recording is comprised is understood by software or system.But, in some large softwares or system,, cause the log information that same incident comprises can not continuous recording because variety of event frequently takes place, each incident is interted record.In this case, owing to do not get in touch with the front and back log information, every log information is isolated relatively, and log analysis tool or log analysis person can not understand the expressed implication of each log information, and work has brought great inconvenience to log analysis.
Summary of the invention
In view of this, fundamental purpose of the present invention is to provide a kind of method and device of handling journal file, use method provided by the present invention and the device can put the incident that writes down in the journal file in order.
For achieving the above object, technical scheme of the present invention is achieved in that
A kind of method of handling journal file, this method comprises:
With the journal file of giving setting analysis is incident under the fundamental analysis log information, and the log information that will belong to same incident is modelling event model storehouse;
According to the event model storehouse of setting up, the journal file of given processing is carried out analysis and arrangement, obtaining with the incident is the journal file of whole record.
A kind of device of handling journal file, this device comprise analytic unit, processing unit and event model storehouse:
Described analytic unit, being used for the journal file of giving setting analysis is incident under the fundamental analysis log information, the log information that will belong to same incident is that model adds the event model storehouse;
Described processing unit is used for the model preserved according to described event model storehouse, and the journal file of given processing is carried out analysis and arrangement, and obtaining with the incident is the journal file of whole record.
The method of processing journal file provided by the present invention and device, by analyzing the model that journal file obtains presentation of events, according to model the journal file of given processing is handled, log information in the journal file is put in order, make the log information of the same incident of expression carry out record with integral body, reflection system or the event of software systems institute that journal file after make handling can be clear, understand make things convenient for carrying out smoothly of log analysis work.The technical scheme that technical scheme of the present invention provides a kind of automatic study, handled automatically possesses versatility, can be applicable to various softwares and system, but also saving manpower that can be bigger.
Description of drawings
Fig. 1 is the exemplary process diagram of the inventive method;
Fig. 2 is the exemplary process diagram of apparatus of the present invention;
Fig. 3 is a process flow diagram of analyzing journal file in the embodiment of the invention;
Fig. 4 is a process flow diagram of handling journal file in the embodiment of the invention;
Fig. 5 is the corresponding relation between event model storehouse and the daily record translation storehouse in the embodiment of the invention.
Embodiment
In order to solve existing problem in the log analysis work, the present invention carries out pre-service to journal file, it is the journal file of unit record that journal file is organized into the incident, make the expression same incident log information recording together so that log analysis tool and log analysis person effectively analyze journal file.
Referring to Fig. 1, Fig. 1 is the exemplary process diagram of the inventive method.This method comprises: in step 101, to give the journal file of setting analysis, analyze the affiliated incident of log information, the log information that will belong to same incident is modelling event model storehouse; In step 102, according to the event model storehouse of setting up, the journal file of given processing is carried out analysis and arrangement, obtaining with the incident is the journal file of whole record.
Here, analyze the process of the affiliated incident of log information, can also can utilize unartificial mode in order to realizing with artificial mode.Wherein, giving the journal file of setting analysis can be the journal file that belongs to same software or system log (SYSLOG) with the journal file of given processing, analyzes the event model that obtains like this and has more specific aim.
Adopting manual type to set up the event model storehouse is the experience that log analysis person relies on self, analyzes the included log information of incident, incident that occurs in this journal file from give the journal file of setting analysis.When adopting unartificial mode to set up the event model storehouse, can be in the following way.
This mode can comprise the steps: A1, selects a log information in giving the journal file of setting analysis.Here, select article one log information in the journal file as benchmark usually.B1, be benchmark, calculate itself and the degree of correlation of M bar log information thereafter respectively with the log information selected; The degree of correlation is integrated with the current log information of analyzing greater than the log information of the threshold value that is provided with; And be benchmark with the integral body after merging, calculate itself and the degree of correlation of M bar log information thereafter respectively, the degree of correlation is integrated with this integral body greater than the log information of threshold value; With the integral body that merges once more is benchmark, calculates itself and the degree of correlation of M bar log information thereafter respectively, and the degree of correlation is integrated with this integral body greater than the log information of threshold value; When not having log information to merge, the integral body that merging is obtained adds the event model storehouse as an event model again.C1, select another not analyze and do not add the log information of model in the journal file of analyzing fixed giving, there is not the log information of not analyzing and do not add model in execution in step B1 in giving the journal file of setting analysis.
Wherein, the degree of correlation is represented the common frequency that occurs of two log informations and integral body and the common frequency that occurs of log information, the analysis bar number of M for being provided with.Owing to belong to the log information of same incident, should be continuous recording usually, but owing to wherein interting the log information that has write down other incidents, if therefore two frequent common appearance of log information, then they represent the probability of same incident just high more.Therefore, in above-mentioned implementation, with the log information of the degree of correlation greater than setting threshold, and log information and whole merge as a wholely, can be used for searching the incident that obtains.
Wherein, the specific implementation that threshold value is set can be: select to belong to the log information of same incident in giving the journal file of setting analysis, calculate the degree of correlation between each log information, the mean value of the degree of correlation is set to described threshold value.Here the log information of selecting to belong to same incident can rely on the mode of artificial selection.The size that threshold value certainly, also can the dependence experience be set.
The computing method of the degree of correlation are between the integral body of foregoing description and the log information: with each member in the integral body be benchmark calculate respectively and log information between the degree of correlation, with the maximal correlation degree that obtains as the degree of correlation between this integral body and the log information.And the concrete log information and the degree of correlation between the log information can foundations
Calculate.Wherein,
Be the degree of correlation that calculates; K and η adjustment parameter for being provided with; α is the inverse of two capable difference mean values of log information; β is C/A+B, C/A+C/B, C/A or C/B, the number of times of A wherein in giving the journal file of setting analysis, occurring as the log information of benchmark, the number of times that B occurs in giving the journal file of setting analysis for another log information, the number of times that C occurs within the M bar log information after as the log information of benchmark for this another log information.
Suppose that current giving in the journal file of setting analysis comprises 50 log informations altogether, certain most journal files comprise the log information greater than 50.Suppose that log information X as benchmark appears at the 2nd, 9,16 in the journal file, and another log information Y appears at the 5th, 13,20 of journal file.Then two capable difference mean value of log information is the ≈ 3.667 of α=(5-2)+(13-9)+(20-16)/3, and Calculation Method is to be the log information Y that benchmark is looked for backward with log information X, and it is poor to calculate row; And then be that benchmark is searched log information Y backward with next log information X, it is poor to calculate row; By that analogy, the capable difference that will calculate is then averaged, and is exactly that the row that needs here to obtain differs from mean value.
Simultaneously, appear at the number of times A=3 of journal file as the log information X of benchmark, another log information Y appears at the number of times B=3 of journal file, suppose that it is 15 that the analysis bar that is provided with is counted M, then the number of times that occurs within back 15 of log information X of log information Y is 2, this moment β=C/A+B=2/3+3 ≈ 0.333.Then the degree of correlation between log information X and the Y is
Wherein, coefficient k and η are mainly used in and regulate the influence that α and β produce the degree of correlation.The influence of α is bigger in some system, and the influence of some β of system is remarkable.The concrete value of k and η can be decided according to concrete system.Usually, β to influence meeting bigger, so k can be smaller than η, recommended value is less than 1: 3.
After an incident is added the event model storehouse as model, can further include: search number of times that this incident occurs altogether and this incident in giving the journal file of setting analysis continuously every the number of times of continuous appearance, when the ratio of the number of times of continuous appearance and total occurrence number more than or equal to be provided with repeat threshold value the time, the log information that this incident comprised is being deleted in giving the journal file of setting analysis; Then, select another not analyze and do not add the log information of model in the journal file of analyzing, set up the operation of event model fixed giving.Wherein, repeat threshold value can rule of thumb be worth be provided with definite.
After having set up the incident model bank, analysis and arrangement is carried out to the journal file of given processing in the event model storehouse that just can utilize foundation, and obtaining with the incident is the journal file of whole record.Concrete can be that reservation original log file is created a journal file in addition, with the order of incident generation, writes down the log information that each incident comprises in the journal file of this part establishment; Also can be on the basis of original log file, to put in order, make journal file carry out record according to incident.Concrete implementation can may further comprise the steps: A2, log information of selection in the journal file of given processing; B2, in the event model storehouse, search the model that obtains comprising selected log information, in the journal file of given processing, search other log informations that whether exist described model to comprise in the back N bar log information of selected log information, if exist, from described journal file, extract the log information that finds, record as a whole; Otherwise, carry out C2; C2, select another untreated log information in the journal file of given processing, there is not untreated log information in execution in step B2 in the journal file of given processing.Wherein, the comparison bar number of N for being provided with.The concrete method that is provided with can rule of thumb be worth definite.
In addition, the Event Description corresponding with incident can also be set; Obtaining with the incident is the journal file of whole record, according to the Event Description that is provided with this journal file is translated, and obtains the Event Description of each incident correspondence.The person effectively analyzes journal file so more to help the log analysis.
In addition, referring to Fig. 2, Fig. 2 is a kind of device of handling journal file provided by the invention, and this device comprises analytic unit, processing unit and event model storehouse.
Wherein, described analytic unit, being used for the journal file of giving setting analysis is incident under the fundamental analysis log information, the log information that will belong to same incident is that model adds the event model storehouse.Described processing unit is used for the model preserved according to described event model storehouse, and the journal file of given processing is carried out analysis and arrangement, and obtaining with the incident is the journal file of whole record.
Particularly, described analytic unit comprises performance element and correlation calculating unit.Described performance element, be used for selecting a log information at the journal file of giving setting analysis, with this log information is that the operation that benchmark is set up model comprises: with the log information selected is benchmark, indicates described correlation calculating unit to calculate itself and the degree of correlation of M bar log information thereafter respectively; Result according to described correlation calculating unit is returned integrates with the current log information of analyzing with the degree of correlation greater than the log information of the threshold value that is provided with; And be benchmark with the integral body after merging, indicate described correlation calculating unit to calculate itself and the degree of correlation of M bar log information thereafter respectively, the result according to described degree of correlation counting unit is returned integrates with this integral body with the degree of correlation greater than the log information of threshold value; With the integral body that merges once more is benchmark, indicates described correlation calculating unit to calculate itself and the degree of correlation of M bar log information thereafter respectively, and the result according to described correlation calculating unit is returned integrates with this integral body with the degree of correlation greater than the log information of threshold value; When not having log information to merge, the integral body that merging is obtained adds the event model storehouse as an event model again; Wherein M is the analysis bar number of setting; Then, select another not analyze and do not add the log information of model in the journal file of analyzing, carry out the operation of setting up model, in giving the journal file of setting analysis, do not have the log information of not analyzing and do not add model fixed giving.Described correlation calculating unit is used for carrying out according to the indication of described performance element the calculating of the degree of correlation, and returns result of calculation to described performance element; The described degree of correlation is represented two log informations and the whole and common frequency that occurs of log information.
In addition, this device further comprises the threshold calculations unit.Described threshold calculations unit, be used for belonging to the log information of same incident in the journal file selection of giving setting analysis, indicate described correlation calculating unit to calculate the degree of correlation between each log information, the result of calculation of returning according to described correlation calculating unit, the mean value of getting the degree of correlation is the threshold value of described setting, uses when merging log information for described performance element.
Wherein, described correlation calculating unit, when the degree of correlation of calculating between integral body and the log information, with each member in the integral body be benchmark calculate respectively and log information between the degree of correlation, with the maximal correlation degree that obtains as the degree of correlation between this integral body and the log information.
The mode that correlation calculating unit is specifically calculated the degree of correlation between log information and the log information be by
Calculate.Wherein,
Be the degree of correlation that calculates; K and η adjustment parameter for being provided with; α is the inverse of two capable difference mean values of log information; β is C/A+B, C/A+C/B, C/A or C/B, the number of times of A wherein in giving the journal file of setting analysis, occurring as the log information of benchmark, the number of times that B occurs in giving the journal file of setting analysis for another log information, the number of times that C occurs within the M bar log information after as the log information of benchmark for this another log information.
In addition, this device further comprises the re-treatment unit.Described re-treatment unit, be used for after the integral body that described performance element obtains merging adds the event model storehouse as an event model, search number of times that described incident occurs altogether and this incident continuously every the number of times of continuous appearance in giving the journal file of setting analysis, when the ratio of the number of times of continuous appearance and total occurrence number more than or equal to be provided with repeat threshold value the time, the log information that this incident comprised is being deleted in giving the journal file of setting analysis.
Described processing unit according to the model of preserving in the described event model storehouse, carries out analysis and arrangement to the journal file of given processing, when obtaining with the incident being the journal file of whole record, selects a log information in the journal file of given processing; In the event model storehouse, search the model that obtains comprising selected log information, in the journal file of given processing, search other log informations that whether exist described model to comprise in the back N bar log information of selected log information, if exist, from described journal file, extract the log information find, and record as a whole; Otherwise, in the journal file of given processing, select another untreated log information, search the affiliated incident of this log information, in the journal file of given processing, there is not untreated log information wherein, the comparison bar number of N for being provided with.
The person effectively analyzes journal file in order further to be the log analysis, and this device can further include translation unit.This translation unit is used to be provided with the Event Description corresponding with incident; And according to the Event Description that is provided with to being that the journal file of whole record is translated with the incident, obtain the Event Description of each incident correspondence.
In order clearly to describe implementation of the present invention, now enumerate embodiment technical scheme of the present invention is done further to describe.
Now hypothesis comprises 500 log informations for the journal file of setting analysis, and it is 20 that the analysis bar of setting is counted M.In the present embodiment, analysis comprises to the process of the journal file of setting analysis:
In step 301, in giving the journal file of setting analysis, select article one log information.Certainly, when selecting log information, also can select non-article one log information.
In step 302, be benchmark with selected log information, calculate itself and the degree of correlation of 20 log informations thereafter respectively.
When selected log information is the 1st log information, then be to calculate the 1st log information respectively with the 2nd, 3,4 ..., 20 the degree of correlation.Wherein, the computing method of the degree of correlation can be according to the formula of introducing above between two log informations
Carry out.
In step 303, according to result calculated, judge whether to exist the log information of the degree of correlation greater than threshold value, if, execution in step 304; Otherwise, execution in step 308.
In step 304, that the degree of correlation is as a whole with the log information merging as benchmark greater than the log information of threshold value.
Here, suppose that the 1st, 2 and 6 log information is the log information that belongs to same incident, the degree of correlation between the 1st and 2 log information and the 1st and the 6th log information will be greater than being provided with threshold value so, and then merge as a whole this moment with the 1st, 2 and 6.
In step 305, calculate to merge the integral body that obtains and the degree of correlation between 20 log informations thereafter.
Here, in order more clearly to be described, thereafter 20 log informations are called the log information of comparing.The computing method of the degree of correlation are between the whole and log information of comparing, with each log information in the integral body be benchmark calculate respectively and the log information of comparing between the degree of correlation, with the maximal correlation degree that obtains as the degree of correlation between this integral body and the log information of comparing.If this degree of correlation then adds this integral body with this daily record greater than the threshold value that is provided with.
In the hypothesis of step 304, the 1st, 2 and 6 log information merged as a wholely, herein, then be to calculate this integral body and the 3rd, 4,5,7,8 ..., the degree of correlation between 22 log informations.
In step 306, according to result calculated, judge whether to exist the log information of the degree of correlation greater than threshold value, if exist, then execution in step 307; Otherwise, the integral body that participates in the step 305 calculating is added event model storehouse, execution in step 308 as model.
In this step, integral body is added after the event model storehouse as model, can search number of times that described incident occurs altogether and this incident continuously every the number of times of continuous appearance in giving the journal file of setting analysis, when the ratio of the number of times of continuous appearance and total occurrence number more than or equal to be provided with repeat threshold value the time, show that this incident is for repeating a large amount of incidents that occur, this information is often represented a kind of unusual, therefore the log information that this incident comprised can be deleted in giving the journal file of setting analysis, be used for clearing up journal file, so that follow-uply better analyze.
In step 307, the degree of correlation is integrated with the integral body that participates in calculating in the step 305, execution in step 305 greater than the log information of threshold value.
In step 308, judge whether to exist the log information of not analyzing and do not incorporate into integral body in the journal file of analyzing fixed giving, if select wherein one, execution in step 302; Otherwise, finish current treatment scheme.
Wherein, the selection wherein concrete mode of a log information can be, according to log information of not analyzing and do not incorporate into integral body of the log information select progressively of writing down in the journal file.For example, when the 1st, 2 and 6 log information merged as a whole add the event model storehouse as incident after, next bar log information of selection then is the 3rd.
By above flow process, can analyze the incident that parses wherein to be comprised, and the log information that incident comprised to the journal file of giving setting analysis.Pre-service is carried out to the journal file of given processing in the event model storehouse that like this, just can utilize foundation.Because different system journals varies and the length of the journal file of the setting analysis of giving, might make the analysis result of influence, be left some and analyzed but the log information of not incorporating any incident into.Can revise by artificial mode for these log informations, for example will become the log information of an incident to add event model separately or the incident in the time model storehouse is revised.In addition, when the event model storehouse that generates was analyzed, if find that the event model storehouse that generates is not very perfect, some incident was not analyzed, and perhaps the log information that incident comprised is incomplete.In this case, can consider the threshold value that is provided with is revised.The mode of revising mainly comprises reduction or improves threshold value.
After analysis obtains the event schema storehouse, just can utilize the event model storehouse that obtains that journal file is carried out pre-service, help better journal file to be analyzed.Referring to Fig. 4, the flow process of the pre-service journal file that Fig. 4 embodiment of the invention provides may further comprise the steps:
In step 401, in the journal file of given processing, select article one log information.Certainly, when selecting log information, also can select non-article one log information.
In step 402, in the event model storehouse, search the event model that obtains comprising selected log information, in the journal file of given processing, search other log informations that whether exist described model to comprise in the back N bar log information of selected log information, if, execution in step 403; Otherwise, execution in step 404.
Wherein,, then respectively these two incidents are handled, whether comprised other log informations of incident in the back N bar log information when in the event model storehouse, having found two or more event models when all comprising selected log information.
In step 403, from journal file, extract the log information that finds, record as a whole.The mode of physical record can be to set up a journal file in addition, is used for writing down the order of occurrence recording events by incident.
Here, suppose that the N that is provided with is 10.At selected article one log information is X.The incident of this log information correspondence also comprises other log informations Y and Z.At this moment, search the 2nd～10 log information and judge wherein whether comprised log information Y and Z,, then log information X, Y and Z are extracted from journal file, log information X, Y and Z are made as a whole record if comprised log information Y and Z.Like this, just can solve owing to log information disperses to cause the problem that log information can't the visual representation incident.
When there are two or more in the incident that comprises selected log information, behind selected log information, all found other daily records that incident comprised in the N bar log information simultaneously, then extract the complete the earliest incident that occurs.The incident that for example comprises log information X has two.One except comprising X, also comprise Y and Z; Another also comprises T except comprising X.Suppose that X is article one log information, the N of setting is 10, and Y and Z appear at the 2nd and 6, and T appears at the 8th.At this moment, extract as incident 1,2 and 6 and preserve.The 8th log information T do not dealt with.
In step 404, in the journal file of given processing, judge whether to exist untreated log information, if then execution in step 405; Otherwise, finish current treatment scheme.
In step 405, in the journal file of given processing, select another untreated log information, execution in step 402.
Wherein, the selection wherein concrete mode of a log information can be, according to untreated log information of the log information select progressively of writing down in the journal file.For example, when the 1st, 2 and 6 log information extracted and do as a whole record from the journal file of given processing, next bar log information of selection then is the 3rd log information.
In addition, determine in the present embodiment whether two log informations are identical, whether two daily records of main dependence comparison represent identical content.When comparing, can a daily record be divided into one by one word according to specific decollator, under the common situation, separator is the space, promptly cuts apart according to the space between word.But in some system, do not carry out the interval between word, but, then need to use these symbols to come daily record is cut apart at this moment with "-" or ": " with the space.In addition,, can when cutting apart, specify before generation model by the user, so that accurate generation model as parameter when the special decollator of needs.
Comparison process is each word of comparing in turn in the log information, when the word number of coupling reaches certain ratio, can think that promptly they are daily records of the same incident of expression.As for reaching much ratios, can specify by the user, in general reach 75% and just can think identical daily record.When carrying out the comparison of two words, in the daily record very when common situation word with a numeral, as disk3.In daily record when numeral a kind of auxiliary expression meaning symbol, can neglect these numerals in the detailed process.For example, daily record may 7 12:23 diskl error and may 8 01:10 disk5 error, after neglecting numeral, the affairs of the same type of expression.
By flow process shown in Figure 4, just can carry out pre-service to journal file, it is the journal file of whole record that journal file is organized into the incident, so that subsequent analysis is used, technical solution of the present invention does not limit concrete analysis mode.
In technical scheme of the present invention, the Event Description corresponding with incident can also be set; To being that the whole journal file that writes down is translated with the incident, obtain the Event Description of each incident correspondence according to the Event Description that is provided with.As shown in Figure 5, the Event Description of each incident correspondence of daily record translation storehouse is formed.Analyze daily record for needs, we use the method for introducing above that it is decomposed, and then the content of decomposing are corresponded to the translation storehouse, so just can obtain the translation of this piece daily record.
Can also once operate except the translation daily record: according to the result of model in the model bank and daily record decomposition, counting needs to analyze the number of times that certain operation occurs in the daily record, number of times that certain occurs unusually; According to the result that the model in the model bank and daily record are decomposed, check the operation, the incident that took place before certain unusual appearance, infer the operation that leads to system abnormity with this, and then help the analyzing and positioning of problem, find product defects; And according to operation note statistics user's operating habit, use habit, and then optimization system design etc.
In technical scheme of the present invention, by being model with system decomposition, according to model journal file is carried out pre-service, the reflection system or the event of software systems institute that make journal file after the processing can be clear, understand make things convenient for carrying out smoothly of log analysis work.Be not difficult to find out that from realization flow of the present invention the technical scheme of the invention provides a kind of automatic study, handling automatically possesses versatility, can be applicable to various softwares and system, saving manpower that can be bigger.Especially for the maintainer of enterprises and institutions, safeguard multiple systems simultaneously, use technical scheme of the present invention can save very big human and material resources to greatest extent at needs.And the event model of being set up in the process of maintenance can be preserved, for using next time, with sharing of the accumulation that reaches experience and technology.
Above embodiment only is unrestricted in order to technical scheme of the present invention to be described, although the present invention is had been described in detail with reference to preferred embodiment, those of ordinary skill in the art is to be understood that, can make amendment or replacement on an equal basis to technical scheme of the present invention, and not breaking away from the spirit and scope of technical solution of the present invention, it all should be encompassed in the middle of the claim scope of the present invention.
Claims (16)
1, a kind of method of handling journal file is characterized in that, this method comprises:
With the journal file of giving setting analysis is incident under the fundamental analysis log information, and the log information that will belong to same incident is modelling event model storehouse;
According to the event model storehouse of setting up, the journal file of given processing is carried out analysis and arrangement, obtaining with the incident is the journal file of whole record.
2, method according to claim 1 is characterized in that, is incident under the fundamental analysis log information with the journal file of giving setting analysis, and the log information that will belong to same incident is that modelling event model storehouse is:
A1, in giving the journal file of setting analysis, select a log information;
B1, be benchmark, calculate itself and the degree of correlation of M bar log information thereafter respectively with the log information selected; The degree of correlation is integrated with the current log information of analyzing greater than the log information of the threshold value that is provided with; And be benchmark with the integral body after merging, calculate itself and the degree of correlation of M bar log information thereafter respectively, the degree of correlation is integrated with this integral body greater than the log information of threshold value; With the integral body that merges once more is benchmark, calculates itself and the degree of correlation of M bar log information thereafter respectively, and the degree of correlation is integrated with this integral body greater than the log information of threshold value; When not having log information to merge, the integral body that merging is obtained adds the event model storehouse as an event model again; Wherein, the degree of correlation is represented two log informations and the whole and common frequency that occurs of log information, the analysis bar number of M for being provided with;
C1, select another not analyze and do not add the log information of model in the journal file of analyzing fixed giving, there is not the log information of not analyzing and do not add model in execution in step B1 in giving the journal file of setting analysis.
3, method according to claim 2, it is characterized in that, the described threshold value that is provided with is: select to belong to the log information of same incident in giving the journal file of setting analysis, calculate the degree of correlation between each log information, the mean value of the degree of correlation is set to described threshold value.
4, method according to claim 2 is characterized in that, the degree of correlation between described calculating integral body and the log information is:
With each member in the integral body be benchmark calculate respectively and log information between the degree of correlation, with the maximal correlation degree that obtains as the degree of correlation between this integral body and the log information.
According to claim 2,3 or 4 described methods, it is characterized in that 5, the degree of correlation between described calculating log information and the log information is:
Foundation
Calculate the degree of correlation between log information and the log information;
Wherein,
Be the degree of correlation that calculates; K and η adjustment parameter for being provided with; α is the inverse of two capable difference mean values of log information; β is C/A+B, C/A or C/B, the number of times of A wherein in giving the journal file of setting analysis, occurring as the log information of benchmark, the number of times that B occurs in giving the journal file of setting analysis for another log information, the number of times that C occurs within the M bar log information after as the log information of benchmark for this another log information.
According to claim 2,3 or 4 described methods, it is characterized in that 6, the integral body that merging is obtained adds after the event model storehouse as an event model, further comprises:
Search number of times that described incident occurs altogether and this incident continuously every the number of times of continuous appearance in giving the journal file of setting analysis, when the ratio of the number of times of continuous appearance and total occurrence number more than or equal to be provided with repeat threshold value the time, the log information that this incident comprised is deleted execution in step C1 in giving the journal file of setting analysis.
7, according to arbitrary described method in the claim 1 to 4, it is characterized in that, described according to the event model storehouse of setting up, the journal file of given processing is carried out analysis and arrangement, obtaining with the incident is that the journal file of whole record is:
A2, in the journal file of given processing, select a log information;
B2, in the event model storehouse, search the model that obtains comprising selected log information, in the journal file of given processing, search other log informations that whether exist described model to comprise in the back N bar log information of selected log information, if exist, from described journal file, extract the log information that finds, record as a whole; Otherwise, carry out C2; Wherein, the comparison bar number of N for being provided with;
C2, select another untreated log information in the journal file of given processing, there is not untreated log information in execution in step B2 in the journal file of given processing.
8, according to arbitrary described method in the claim 1 to 4, it is characterized in that this method further comprises:
The Event Description corresponding with incident is set;
To being that the whole journal file that writes down is translated with the incident, obtain the Event Description of each incident correspondence according to the Event Description that is provided with.
9, a kind of device of handling journal file is characterized in that, this device comprises analytic unit, processing unit and event model storehouse:
Described analytic unit, being used for the journal file of giving setting analysis is incident under the fundamental analysis log information, the log information that will belong to same incident is that model adds the event model storehouse;
Described processing unit is used for the model preserved according to described event model storehouse, and the journal file of given processing is carried out analysis and arrangement, and obtaining with the incident is the journal file of whole record.
10, device according to claim 9 is characterized in that, described analytic unit comprises performance element and correlation calculating unit;
Described performance element, be used for selecting a log information at the journal file of giving setting analysis, with this log information is that the operation that benchmark is set up model comprises: with the log information selected is benchmark, indicates described correlation calculating unit to calculate itself and the degree of correlation of M bar log information thereafter respectively; Result according to described correlation calculating unit is returned integrates with the current log information of analyzing with the degree of correlation greater than the log information of the threshold value that is provided with; And be benchmark with the integral body after merging, indicate described correlation calculating unit to calculate itself and the degree of correlation of M bar log information thereafter respectively, the result according to described degree of correlation counting unit is returned integrates with this integral body with the degree of correlation greater than the log information of threshold value; With the integral body that merges once more is benchmark, indicates described correlation calculating unit to calculate itself and the degree of correlation of M bar log information thereafter respectively, and the result according to described correlation calculating unit is returned integrates with this integral body with the degree of correlation greater than the log information of threshold value; When not having log information to merge, the integral body that merging is obtained adds the event model storehouse as an event model again; Wherein M is the analysis bar number of setting; Then, select another not analyze and do not add the log information of model in the journal file of analyzing, carry out the operation of setting up model, in giving the journal file of setting analysis, do not have the log information of not analyzing and do not add model fixed giving;
Described relevant seat computing unit is used for carrying out according to the indication of described performance element the calculating of the degree of correlation, and returns result of calculation to described performance element; The described degree of correlation is represented two log informations and the whole and common frequency that occurs of log information.
11, device according to claim 10 is characterized in that, this device further comprises the threshold calculations unit;
Described threshold calculations unit, be used for belonging to the log information of same incident in the journal file selection of giving setting analysis, indicate described correlation calculating unit to calculate the degree of correlation between each log information, the result of calculation of returning according to described correlation calculating unit, the mean value of getting the degree of correlation is the threshold value of described setting, uses when merging log information for described performance element.
12, device according to claim 10 is characterized in that,
Described correlation calculating unit, when the degree of correlation of calculating between integral body and the log information, with each member in the integral body be benchmark calculate respectively and log information between the degree of correlation, with the maximal correlation degree that obtains as the degree of correlation between this integral body and the log information.
13, according to claim 10,11 or 12 described devices, it is characterized in that,
Described correlation calculating unit, be used for by
Calculate the degree of correlation between log information and the log information; Wherein,
Be the degree of correlation that calculates; K and η adjustment parameter for being provided with; α is the inverse of two capable difference mean values of log information; β is C/A+B, C/A+C/B, C/A or C/B, the number of times of A wherein in giving the journal file of setting analysis, occurring as the log information of benchmark, the number of times that B occurs in giving the journal file of setting analysis for another log information, the number of times that C occurs within the M bar log information after as the log information of benchmark for this another log information.
14, according to the described device of arbitrary claim in the claim 10 to 12, it is characterized in that this device further comprises the re-treatment unit;
Described re-treatment unit, be used for after the integral body that described performance element obtains merging adds the event model storehouse as an event model, search number of times that described incident occurs altogether and this incident continuously every the number of times of continuous appearance in giving the journal file of setting analysis, when the ratio of the number of times of continuous appearance and total occurrence number more than or equal to be provided with repeat threshold value the time, the log information that this incident comprised is being deleted in giving the journal file of setting analysis.
15, according to the described device of arbitrary claim in the claim 9 to 12, it is characterized in that,
Described processing unit is used for selecting a log information at the journal file of given processing; In the event model storehouse, search the model that obtains comprising selected log information, in the journal file of given processing, search other log informations that whether exist described model to comprise in the back N bar log information of selected log information, if exist, from described journal file, extract the log information find, and record as a whole; Otherwise, in the journal file of given processing, select another untreated log information, search the affiliated incident of this log information, in the journal file of given processing, there is not untreated log information; Wherein, the comparison bar number of N for being provided with.
16, according to the described device of arbitrary claim in the claim 9 to 12, it is characterized in that this device further comprises translation unit;
Described translation unit is used to be provided with the Event Description corresponding with incident; And according to the Event Description that is provided with to being that the journal file of whole record is translated with the incident, obtain the Event Description of each incident correspondence.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200810118286A CN100590603C (en) | 2008-08-12 | 2008-08-12 | Method and device for processing log file |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200810118286A CN100590603C (en) | 2008-08-12 | 2008-08-12 | Method and device for processing log file |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101339531A true CN101339531A (en) | 2009-01-07 |
| CN100590603C CN100590603C (en) | 2010-02-17 |
Family
ID=40213602
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200810118286A Active CN100590603C (en) | 2008-08-12 | 2008-08-12 | Method and device for processing log file |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN100590603C (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102208991A (en) * | 2010-03-29 | 2011-10-05 | 腾讯科技（深圳）有限公司 | Blog processing method, device and system |
| CN102902764A (en) * | 2012-09-25 | 2013-01-30 | 北京奇虎科技有限公司 | Method and device for log recording |
| CN103425568A (en) * | 2013-08-23 | 2013-12-04 | 新浪网技术(中国)有限公司 | Method and device for processing log information |
| CN103929321A (en) * | 2013-01-15 | 2014-07-16 | 腾讯科技（深圳）有限公司 | Log processing method and device |
| CN104166563A (en) * | 2014-08-11 | 2014-11-26 | Tcl通讯（宁波）有限公司 | Method and system for controlling repeatedly output logs based on mobile terminal |
| CN104239475A (en) * | 2014-09-03 | 2014-12-24 | 北京优特捷信息技术有限公司 | Method and device for analyzing time series data |
| CN104572781A (en) * | 2013-10-29 | 2015-04-29 | 中国银联股份有限公司 | Method and device for generating transaction log |
| CN106294406A (en) * | 2015-05-22 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus accessing data for processing application |
| CN106502875A (en) * | 2016-10-21 | 2017-03-15 | 过冬 | A kind of daily record generation method and system based on cloud computing |
| CN106528566A (en) * | 2015-09-11 | 2017-03-22 | 北京国双科技有限公司 | Log file output method, server and client |
| CN106933925A (en) * | 2015-12-31 | 2017-07-07 | 北京国双科技有限公司 | The method and apparatus for obtaining click behavior |
-
2008
- 2008-08-12 CN CN200810118286A patent/CN100590603C/en active Active
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102208991A (en) * | 2010-03-29 | 2011-10-05 | 腾讯科技（深圳）有限公司 | Blog processing method, device and system |
| CN102902764A (en) * | 2012-09-25 | 2013-01-30 | 北京奇虎科技有限公司 | Method and device for log recording |
| CN102902764B (en) * | 2012-09-25 | 2016-05-11 | 北京奇虎科技有限公司 | A kind of method and apparatus of log recording |
| CN103929321A (en) * | 2013-01-15 | 2014-07-16 | 腾讯科技（深圳）有限公司 | Log processing method and device |
| CN103425568B (en) * | 2013-08-23 | 2016-08-10 | 新浪网技术(中国)有限公司 | log information processing method and device |
| CN103425568A (en) * | 2013-08-23 | 2013-12-04 | 新浪网技术(中国)有限公司 | Method and device for processing log information |
| CN104572781B (en) * | 2013-10-29 | 2018-10-23 | 中国银联股份有限公司 | A kind of transaction log production method and device |
| CN104572781A (en) * | 2013-10-29 | 2015-04-29 | 中国银联股份有限公司 | Method and device for generating transaction log |
| CN104166563A (en) * | 2014-08-11 | 2014-11-26 | Tcl通讯（宁波）有限公司 | Method and system for controlling repeatedly output logs based on mobile terminal |
| CN104166563B (en) * | 2014-08-11 | 2017-12-12 | Tcl通讯（宁波）有限公司 | The method and system being controlled based on mobile terminal to the log for repeating output |
| CN104239475A (en) * | 2014-09-03 | 2014-12-24 | 北京优特捷信息技术有限公司 | Method and device for analyzing time series data |
| CN106294406A (en) * | 2015-05-22 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of method and apparatus accessing data for processing application |
| CN106294406B (en) * | 2015-05-22 | 2020-04-17 | 阿里巴巴集团控股有限公司 | Method and equipment for processing application access data |
| CN106528566A (en) * | 2015-09-11 | 2017-03-22 | 北京国双科技有限公司 | Log file output method, server and client |
| CN106933925A (en) * | 2015-12-31 | 2017-07-07 | 北京国双科技有限公司 | The method and apparatus for obtaining click behavior |
| CN106502875A (en) * | 2016-10-21 | 2017-03-15 | 过冬 | A kind of daily record generation method and system based on cloud computing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN100590603C (en) | 2010-02-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100590603C (en) | Method and device for processing log file | |
| Fournier | Evaluating text segmentation using boundary edit distance | |
| CN102915335B (en) | Based on the information correlation method of user operation records and resource content | |
| US7788263B2 (en) | Probabilistic retrospective event detection | |
| CN103761264B (en) | Concept hierarchy establishing method based on product review document set | |
| CN104700190B (en) | One kind is for project and the matched method and apparatus of professional | |
| CN102662960A (en) | On-line supervised theme-modeling and evolution-analyzing method | |
| CN102708100A (en) | Method and device for digging relation keyword of relevant entity word and application thereof | |
| CN108446305A (en) | The system and method for various dimensions service data statistics | |
| Khan et al. | Data tweening: incremental visualization of data transforms | |
| CN103020159A (en) | Method and device for news presentation facing events | |
| US10489266B2 (en) | Generating a visualization of a metric at one or multiple levels of execution of a database workload | |
| Kogilavani et al. | Clustering and feature specific sentence extraction based summarization of multiple documents | |
| CN101887415B (en) | Automatic extraction method for text document theme word meaning | |
| CN102053978A (en) | Method and device for extracting subject term from simple sentence | |
| Hand | What is the purpose of statistical modelling | |
| Kogilavani et al. | Clustering based optimal summary generation using genetic algorithm | |
| Xu et al. | HLTCOE at TREC 2013: Temporal Summarization. | |
| CN110874366A (en) | Data processing and query method and device | |
| CN116484084B (en) | Metadata blood-margin analysis method, medium and system based on application information mining | |
| CN102222119A (en) | Automatic personalized abstracting method in digital library system | |
| KR101613397B1 (en) | Method and apparatus for associating topic data with numerical time series | |
| JP5436356B2 (en) | Period-specific subject phrase extraction apparatus, method, and program | |
| JP4892896B2 (en) | Communication analysis apparatus and method | |
| Mirza et al. | Switch detector: an activity spotting system for desktop |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 310052 Binjiang District Changhe Road, Zhejiang, China, No. 466, No. Patentee after: Xinhua three Technology Co., Ltd. Address before: 310053 Hangzhou hi tech Industrial Development Zone, Zhejiang province science and Technology Industrial Park, No. 310 and No. six road, HUAWEI, Hangzhou production base Patentee before: Huasan Communication Technology Co., Ltd. |