CN106254109A - Log collection method, Log Collect System and server - Google Patents
Log collection method, Log Collect System and server Download PDFInfo
- Publication number
- CN106254109A CN106254109A CN201610623314.6A CN201610623314A CN106254109A CN 106254109 A CN106254109 A CN 106254109A CN 201610623314 A CN201610623314 A CN 201610623314A CN 106254109 A CN106254109 A CN 106254109A
- Authority
- CN
- China
- Prior art keywords
- log
- log information
- information
- configuration file
- collection point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
Abstract
The invention provides a kind of log collection method, Log Collect System and server, wherein, log collection method includes: isolate target collection point according to the identification information of the collection point of server；According to default configuration file, the log information that isolation collection point generates is combined to form information object set；According to default configuration file, information object set is converted to specified format, and the information object set of specified format is stored to the JFS of server.Pass through technical solution of the present invention, it is achieved that log information is carried out shunting process, improve the work efficiency of server, add motility and accuracy that log information gathers, decrease the interference of unrelated log information, improve the analysis efficiency to log information.
Description
Technical field
The present invention relates to server technology field, in particular to a kind of log collection method, a kind of log collection system
System and a kind of server.
Background technology
In the related, the log collection point of operation system (user terminal/server framework pattern) need to pre-set, by
Can break down wherein in operation system cannot be predicted, it is therefore desirable to collection point is the most all set, thus
Cause collection point to be compared and be dispersed without the regularity of distribution, when operation system operation troubles, when needing to analyze in time, due to valuable
Daily record flooded by the most unrelated daily record, reduce problem investigation efficiency.
Therefore, how to design a kind of new log collection scheme, work during to improve in daily record location and to analyze fault
Efficiency becomes technical problem urgently to be resolved hurrily.
Summary of the invention
The present invention is based at least one above-mentioned technical problem, it is proposed that a kind of new log collection scheme, by root
According to the identification information of collection point, collection point is isolated, according to default configuration file, the log information that isolation collection point generates is entered
Row assembles information object set, and is converted to specified format, to store to the JFS of server, it is achieved that to day
Will information carries out shunting process, improves the work efficiency of server, adds motility and accuracy that log information gathers,
Decrease the interference of unrelated log information, improve the analysis efficiency to log information.
In view of this, the present invention proposes a kind of log collection method, including: according to the mark letter of the collection point of server
Target collection point is isolated by breath；Combine to be formed by the log information that isolation collection point generates according to default configuration file
Information object set；According to default configuration file, information object set is converted to specified format, and by the letter of specified format
Breath object set stores to the JFS of server.
In this technical scheme, by the identification information according to collection point, collection point is isolated, according to default configuration literary composition
The log information that isolation collection point generates is carried out assembling information object set by part, and is converted to specified format, to store to clothes
In the JFS of business device, it is achieved that log information is carried out shunting process, improves the work efficiency of server, increase
Motility that log information gathers and accuracy, decrease the interference of unrelated log information, improve and divide log information
Analysis efficiency.
Specifically, according to the identification information of the collection point of server, target collection point is isolated, target collection point pair
The region that server easily breaks down carries out log information collection, target collection point is carried out isolation can by communication port every
From or Network Isolation platform realize, also will the data isolation communication of log information and other collection points of target collection point, also
I.e. while the collection of log information, i.e. achieve the isolation of preliminary log information, reduce the dry of unrelated log information
Disturb, improve accuracy and the analysis efficiency of user of daily record location.
Further, the log information of isolation collection point is combined to form information object set, log information is adopted
Collection is similar to pipeline thread, log information is read from each collection point isolated according to the acquisition parameter in default configuration file
Take out, according to the requirement of user, the log information of the collection point of isolation can be diverted in different JFSs,
Make in server the division of labor between each device definitely, and the log information of different acquisition point can be carried out by shunting
Classification, is polymerized the log information of same collection point, enables log information precisely to store, reduce unrelated log information
Interference, improve daily record location accuracy and the analysis efficiency of user.
It addition, information object set is converted to specified format according to the configuration file preset, and carry out with specified format
Storage, the tissue making log information is more flexible, and it is more convenient to process.
Log information is the information of hard disk, software and system problem in record system, simultaneously can be to occur in monitoring system
Event, such as include application log, security log, system journal and web administration daily record etc..
Such as, isolation collection point A, isolation collection point B and isolation collection point C, it is respectively used to the A service area of acquisition server
Fisrt fault log information, the second fault log information of B service area of server and server C service area the 3rd therefore
Barrier log information, according to acquisition time by Fisrt fault log information, the second fault log information and the 3rd fault log information
Being stored in application stack with key assignments form, and combine to be formed information object set, each information object set can will be adopted
The collection time period is as mark, and then can determine collection corresponding when breaking down when any of the above-described server fail
Time period, and the information object set of correspondence, improve accident analysis efficiency and the accuracy rate of log information, decrease computing
Amount.
In technique scheme, it is preferable that target collection is clicked on by the identification information in the collection point according to server
Before row isolation, also including: the acquisition parameter of preset configuration file, wherein, acquisition parameter includes specified format, log information class
In the store path of trigger condition, log information collection period and information object set that type, log information gather at least one
Kind.
In this technical scheme, by the acquisition parameter of preset configuration file, it is achieved that the flexible storage of log information, with
Time ensure that the effectiveness that log information gathers, acquisition parameter includes what specified format, log information type, log information gathered
Trigger condition, log information gather at least one of the store path of period and information object set, by default acquisition parameter,
The log information of different acquisition point can be diverted in different JFSs.
Specifically, it is intended that form includes that W3C (World Wide Web Consortium, World Wide Web Consortium) extends daily record
File format (its encoded content is defaulted as ASCII text), Apache access log form, ArcSight public accident form with
And IDMEF (Intrusion of based on XML (Extensive Makeup Language, extensible markup language)
Detection Message Exchange Format, message interchange format) etc. form, and according to the application program performed not
With, log information type is the most different, such as performs process logs, global positioning system (Global according to test plan
Positioning System, GPS) location information log and call information daily record etc., log information type can also be according to
The concrete executive mode such as debugging, warning, information, mistake makes a distinction.Gathering the period for log information can be according to fixing test
Time indicates.
In any of the above-described technical scheme, it is preferable that day isolation collection point generated according to default configuration file
Will information combines to form information object set, specifically includes following steps: the log information that collection point generates is isolated in inquiry；Root
According to default configuration file, obtain the log information corresponding with log information type；Combine to be formed by the log information obtained
Information object set.
In this technical scheme, the daily record corresponding with the log information type preset by inquiry log information acquisition is believed
Breath, with combination to form information object set, it is achieved that the polymerization to similar log information, enters the log information of same format
Row batch processing, improves the work efficiency of server.
Specifically, combining log information to form information object set according to default configuration file, configuration file is used
In describing the assembling form of log information, such as according to default configuration file by log information with corresponding application name
The forms such as title, log information size, log information acquisition time and log information content assemble.
In any of the above-described technical scheme, it is preferable that also include: information object set is stored in the form of key assignments
In the application stack of server.
In this technical scheme, by information object set to be stored in the application stack of server with the form of key assignments
In, it is achieved that editor and the lookup to the log information in information object set.
Specifically, key assignments is positioned at registration table structural chain end, comprises the actual disposition information used when application program performs
And data, generally include several data type (such as character string, binary value, DWORD value etc.), to adapt to the use of varying environment
Demand, according to key assignments can be easily accomplished in the application stack to server storage the increase of data of log information, deletion,
The operation such as amendment and inquiry.
Wherein, the storage operation of key assignments (key-value in distributed data system) form includes:
(1) perform tree search and insert the position at data (i.e. the data of log information) place with location；
(2) father node of this position is locked；
(3) new leaf node is created；
(4) being write by leaf node, this write operation occurs in internal memory (application stack of the application), and returns one
Individual numerical value, this numerical value determines leaf node and writes the position of hard disk；
(5) amendment father node points to quoting of this leaf node, and this father node had both held the leaf node of sensing internal memory
Quote and hold the numerical value of position of disk；
(6) this father node of labelling is " dirty " (data meaning in internal memory do not appear in disk)；
(7) unlock this father node, by node inspection operation, the data of the log information of father node sensing are written to firmly
Dish.
Inquiry with key assignments form is operated and includes:
Navigate to store the leaf node of the structure tree of log information, inquired about the number reading log information by major key
According to.
In any of the above-described technical scheme, it is preferable that also include: obtaining user, the renewal of configuration file is instructed
Time, the configuration file after updating replaces with default configuration file.
In this technical scheme, the configuration file after updating replaces with default configuration file, start by set date pipe line
Journey, all can read up-to-date configuration file when pipeline thread starts every time, thus the shunting gatherer process of Update log information, carry
Rise the motility that log information gathers.
According to a second aspect of the present invention, it is also proposed that a kind of Log Collect System, including: isolated location, for according to clothes
Target collection point is isolated by the identification information of the collection point of business device；Module units, the configuration file preset for basis will
The log information that isolation collection point generates combines to form information object set；Memory element, for according to the configuration literary composition preset
Information object set is converted to specified format by part, and the information object set of specified format stores the daily record literary composition to server
In part system.
In this technical scheme, by the identification information according to collection point, collection point is isolated, according to default configuration literary composition
The log information that isolation collection point generates is combined forming information object set by part, and is converted to specified format, to deposit
Store up to the JFS of server, it is achieved that log information is carried out shunting process, improve the work effect of server
Rate, adds motility and accuracy that log information gathers, decreases the interference of unrelated log information, improve and believe daily record
The analysis efficiency of breath.
Specifically, according to the identification information of the collection point of server, target collection point is isolated, target collection point pair
The region that server easily breaks down carries out log information collection, target collection point is carried out isolation can by communication port every
From or Network Isolation platform realize, also will the data isolation communication of log information and other collection points of target collection point, also
I.e. while the collection of log information, i.e. achieve the isolation of preliminary log information, reduce the dry of unrelated log information
Disturb, improve accuracy and the analysis efficiency of user of daily record location.
Further, the log information of isolation collection point is combined to form information object set, log information is adopted
Collection is similar to pipeline thread, log information is read from each collection point isolated according to the acquisition parameter in default configuration file
Take out, according to the requirement of user, the log information of the collection point of isolation can be diverted in different JFSs,
Make in server the division of labor between each device definitely, and the log information of different acquisition point can be carried out by shunting
Classification, is polymerized the log information of same collection point, enables log information precisely to store, reduce unrelated log information
Interference, improve daily record location accuracy and the analysis efficiency of user.
It addition, information object set is converted to specified format according to the configuration file preset, and carry out with specified format
Storage, the tissue making log information is more flexible, and it is more convenient to process.
Log information is the information of hard disk, software and system problem in record system, simultaneously can be to occur in monitoring system
Event, such as include application log, security log, system journal and web administration daily record etc..
Such as, isolation collection point A, isolation collection point B and isolation collection point C, it is respectively used to the A service area of acquisition server
Fisrt fault log information, the second fault log information of B service area of server and server C service area the 3rd therefore
Barrier log information, according to acquisition time by Fisrt fault log information, the second fault log information and the 3rd fault log information
Being stored in application stack with key assignments form, and combine to be formed information object set, each information object set can will be adopted
The collection time period is as mark, and then can determine collection corresponding when breaking down when any of the above-described server fail
Time period, and the information object set of correspondence, improve accident analysis efficiency and the accuracy rate of log information, decrease computing
Amount.
In technique scheme, it is preferable that also include: preset unit, for the acquisition parameter of preset configuration file, its
In, acquisition parameter includes that trigger condition that specified format, log information type, log information gather, log information gather the period
With at least one in the store path of information object set.
In this technical scheme, by the acquisition parameter of preset configuration file, it is achieved that the flexible storage of log information, with
Time ensure that the effectiveness that log information gathers, acquisition parameter includes what specified format, log information type, log information gathered
Trigger condition, log information gather at least one of the store path of period and information object set, by default acquisition parameter,
The log information of different acquisition point can be diverted in different JFSs.
Specifically, it is intended that form includes that W3C extension log file formats, Apache access log form, ArcSight are public
The form such as event format and IDMEF based on XML altogether, and different according to the application program performed, and log information type is the most not
With, such as determine according to test plan execution process logs, global positioning system (Global Positioning System, GPS)
Position information log and call information daily record etc., log information type can also be concrete according to debugging, warning, information, mistake etc.
Executive mode makes a distinction.Can indicate according to the fixing testing time for the log information collection period.Any of the above-described skill
In art scheme, it is preferable that also include: query unit, for inquiring about the log information that isolation collection point generates；Acquiring unit, uses
According to the configuration file preset, obtain the log information corresponding with log information type；Module units is additionally operable to: by obtain
Log information combines to form information object set.
In this technical scheme, the daily record corresponding with the log information type preset by inquiry log information acquisition is believed
Breath, with combination to form information object set, it is achieved that the polymerization to similar log information, enters the log information of same format
Row batch processing, improves the work efficiency of server.
Specifically, combining log information to form information object set according to default configuration file, configuration file is used
In describing the assembling form of log information, such as according to default configuration file by log information with corresponding application name
The forms such as title, log information size, log information acquisition time and log information content assemble.
In any of the above-described technical scheme, it is preferable that memory element is additionally operable to: by information object set with the shape of key assignments
Formula is stored in the application stack of server.
In this technical scheme, by information object set to be stored in the application stack of server with the form of key assignments
In, it is achieved that editor and the lookup to the log information in information object set.
Specifically, key assignments is positioned at registration table structural chain end, comprises the actual disposition information used when application program performs
And data, generally include several data type (such as character string, binary value, DWORD value etc.), to adapt to the use of varying environment
Demand, according to key assignments can be easily accomplished in the application stack to server storage the increase of data of log information, deletion,
The operation such as amendment and inquiry.
Wherein, the storage operation of key assignments (key-value in distributed data system) form includes:
(1) perform tree search and insert the position at data (i.e. the data of log information) place with location；
(2) father node of this position is locked；
(3) new leaf node is created；
(4) being write by leaf node, this write operation occurs in internal memory (application stack of the application), and returns one
Individual numerical value, this numerical value determines leaf node and writes the position of hard disk；
(5) amendment father node points to quoting of this leaf node, and this father node had both held the leaf node of sensing internal memory
Quote and hold the numerical value of position of disk；
(6) this father node of labelling is " dirty " (data meaning in internal memory do not appear in disk)；
(7) unlock this father node, by node inspection operation, the data of the log information of father node sensing are written to firmly
Dish.
Inquiry with key assignments form is operated and includes:
Navigate to store the leaf node of the structure tree of log information, inquired about the number reading log information by major key
According to.
In any of the above-described technical scheme, it is preferable that also include: updating block, for obtaining user to configuration literary composition
When the renewal of part instructs, the configuration file after updating replaces with default configuration file.
In this technical scheme, the configuration file after updating replaces with default configuration file, start by set date pipe line
Journey, all can read up-to-date configuration file when pipeline thread starts every time, thus the shunting gatherer process of Update log information, carry
Rise the motility that log information gathers.
According to a third aspect of the present invention, it is also proposed that a kind of server, including the day described in any of the above-described technical scheme
Will acquisition system, therefore, this server has the technology identical with the Log Collect System described in any of the above-described technical scheme
Effect, does not repeats them here.
By above technical scheme, by the identification information according to collection point, collection point is isolated, according to default configuration
The log information that isolation collection point generates is carried out assembling and being converted to specified format by file, stores the journal file to server
In system, it is achieved that log information is carried out shunting process, improve the work efficiency of server, add log information collection
Motility and accuracy, decrease the interference of unrelated log information, improve the analysis efficiency to log information.
Accompanying drawing explanation
Fig. 1 shows the schematic flow diagram of log collection method according to an embodiment of the invention；
Fig. 2 shows the schematic block diagram of Log Collect System according to an embodiment of the invention；
Fig. 3 shows the schematic block diagram of terminal according to an embodiment of the invention；
Fig. 4 shows the schematic flow diagram of log collection method according to another embodiment of the invention；
Fig. 5 shows the schematic block diagram of Log Collect System according to another embodiment of the invention.
Detailed description of the invention
In order to be more clearly understood that the above-mentioned purpose of the present invention, feature and advantage, real with concrete below in conjunction with the accompanying drawings
The present invention is further described in detail by mode of executing.It should be noted that in the case of not conflicting, the enforcement of the application
Feature in example and embodiment can be mutually combined.
Elaborate a lot of detail in the following description so that fully understanding the present invention, but, the present invention also may be used
Implementing to use third party to be different from third party's mode described here, therefore, protection scope of the present invention is not by following
The restriction of disclosed specific embodiment.
Fig. 1 shows the schematic flow diagram of log collection method according to an embodiment of the invention.
As it is shown in figure 1, method according to an embodiment of the invention, including: step 102, according to the collection of server
Target collection point is isolated by the identification information of point；Step 104, generates isolation collection point according to default configuration file
Log information combines to form information object set；Step 106, is converted to information object set according to default configuration file
Specified format, and the information object set of specified format is stored to the JFS of server.
In this technical scheme, by the identification information according to collection point, collection point is isolated, according to default configuration literary composition
The log information that isolation collection point generates is combined forming information object set by part, and is converted to specified format, to deposit
Store up to the JFS of server, it is achieved that log information is carried out shunting process, improve the work effect of server
Rate, adds motility and accuracy that log information gathers, decreases the interference of unrelated log information, improve and believe daily record
The analysis efficiency of breath.
Specifically, according to the identification information of the collection point of server, target collection point is isolated, target collection point pair
The region that server easily breaks down carries out log information collection, target collection point is carried out isolation can by communication port every
From or Network Isolation platform realize, also will the data isolation communication of log information and other collection points of target collection point, also
I.e. while the collection of log information, i.e. achieve the isolation of preliminary log information, reduce the dry of unrelated log information
Disturb, improve accuracy and the analysis efficiency of user of daily record location.
Further, the log information of isolation collection point is combined to form information object set, log information is adopted
Collection is similar to pipeline thread, log information is read from each collection point isolated according to the acquisition parameter in default configuration file
Take out, according to the requirement of user, the log information of the collection point of isolation can be diverted in different JFSs,
Make in server the division of labor between each device definitely, and the log information of different acquisition point can be carried out by shunting
Classification, is polymerized the log information of same collection point, enables log information precisely to store, reduce unrelated log information
Interference, improve daily record location accuracy and the analysis efficiency of user.
It addition, information object set is converted to specified format according to the configuration file preset, and carry out with specified format
Storage, the tissue making log information is more flexible, and it is more convenient to process.
Log information is the information of hard disk, software and system problem in record system, simultaneously can be to occur in monitoring system
Event, such as include application log, security log, system journal and web administration daily record etc..
Such as, isolation collection point A, isolation collection point B and isolation collection point C, it is respectively used to the A service area of acquisition server
Fisrt fault log information, the second fault log information of B service area of server and server C service area the 3rd therefore
Barrier log information, according to acquisition time by Fisrt fault log information, the second fault log information and the 3rd fault log information
Being stored in application stack with key assignments form, and combine to be formed information object set, each information object set can will be adopted
The collection time period is as mark, and then can determine collection corresponding when breaking down when any of the above-described server fail
Time period, and the information object set of correspondence, improve accident analysis efficiency and the accuracy rate of log information, decrease computing
Amount.
In technique scheme, it is preferable that target collection is clicked on by the identification information in the collection point according to server
Before row isolation, also including: the acquisition parameter of preset configuration file, wherein, acquisition parameter includes specified format, log information class
In the store path of trigger condition, log information collection period and information object set that type, log information gather at least one
Kind.
In this technical scheme, by the acquisition parameter of preset configuration file, it is achieved that the flexible storage of log information, with
Time ensure that the effectiveness that log information gathers, acquisition parameter includes what specified format, log information type, log information gathered
Trigger condition, log information gather at least one of the store path of period and information object set, by default acquisition parameter,
The log information of different acquisition point can be diverted in different JFSs.
Specifically, it is intended that form includes that W3C extension log file formats, Apache access log form, ArcSight are public
The form such as event format and IDMEF based on XML altogether.And different according to the application program performed, log information type is the most not
With, such as determine according to test plan execution process logs, global positioning system (Global Positioning System, GPS)
Position information log and call information daily record etc., log information type can also be concrete according to debugging, warning, information, mistake etc.
Executive mode makes a distinction.Can indicate according to the fixing testing time for the log information collection period.
In any of the above-described technical scheme, it is preferable that day isolation collection point generated according to default configuration file
Will information combines to form information object set, specifically includes following steps: the log information that collection point generates is isolated in inquiry；Root
According to default configuration file, obtain the log information corresponding with log information type；Combine to be formed by the log information obtained
Information object set.
In this technical scheme, the daily record corresponding with the log information type preset by inquiry log information acquisition is believed
Breath, with combination to form information object set, it is achieved that the polymerization to similar log information, enters the log information of same format
Row batch processing, improves the work efficiency of server.
Specifically, key assignments is positioned at registration table structural chain end, comprises the actual disposition information used when application program performs
And data, generally include several data type (such as character string, binary value, DWORD value etc.), to adapt to the use of varying environment
Demand, according to key assignments can be easily accomplished in the application stack to server storage the increase of data of log information, deletion,
The operation such as amendment and inquiry.
Wherein, the storage operation of key assignments (key-value in distributed data system) form includes:
(1) perform tree search and insert the position at data (i.e. the data of log information) place with location；
(2) father node of this position is locked；
(3) new leaf node is created；
(4) being write by leaf node, this write operation occurs in internal memory (application stack of the application), and returns one
Individual numerical value, this numerical value determines leaf node and writes the position of hard disk；
(5) amendment father node points to quoting of this leaf node, and this father node had both held the leaf node of sensing internal memory
Quote and hold the numerical value of position of disk；
(6) this father node of labelling is " dirty " (data meaning in internal memory do not appear in disk)；
(7) unlock this father node, by node inspection operation, the data of the log information of father node sensing are written to firmly
Dish.
Inquiry with key assignments form is operated and includes:
Navigate to store the leaf node of the structure tree of log information, inquired about the number reading log information by major key
According to.
In any of the above-described technical scheme, it is preferable that also include: information object set is stored in the form of key assignments
In the application stack of server.
In this technical scheme, by information object set to be stored in the application stack of server with the form of key assignments
In, it is achieved that editor and the lookup to the log information in information object set.
Specifically, key assignments is positioned at registration table structural chain end, comprises the actual disposition information used when application program performs
And data, generally include several data type, to adapt to the use demand of varying environment, can be easily accomplished clothes according to key assignments
The increase of the data of the log information of storage in the application stack of business device, delete, revise and the operation such as inquiry.
In any of the above-described technical scheme, it is preferable that also include: obtaining user, the renewal of configuration file is instructed
Time, the configuration file after updating replaces with default configuration file.
In this technical scheme, the configuration file after updating replaces with default configuration file, start by set date pipe line
Journey, all can read up-to-date configuration file when pipeline thread starts every time, thus the shunting gatherer process of Update log information, carry
Rise the motility that log information gathers.
Fig. 2 shows the schematic block diagram of Log Collect System according to an embodiment of the invention.
As in figure 2 it is shown, system 200 according to an embodiment of the invention, including: isolated location 202, for according to clothes
Target collection point is isolated by the identification information of the collection point of business device；Module units 204, for according to the configuration file preset
The log information that isolation collection point generates is combined to form information object set；Memory element 206, for according to joining of presetting
Put file and information object set is converted to specified format, and the information object set of specified format is stored the day to server
In will file system.
In this technical scheme, by the identification information according to collection point, collection point is isolated, according to default configuration literary composition
The log information that isolation collection point generates is combined forming information object set by part, and is converted to specified format, to deposit
Store up to the JFS of server, it is achieved that log information is carried out shunting process, improve the work effect of server
Rate, adds motility and accuracy that log information gathers, decreases the interference of unrelated log information, improve and believe daily record
The analysis efficiency of breath.
Specifically, according to the identification information of the collection point of server, target collection point is isolated, target collection point pair
The region that server easily breaks down carries out log information collection, target collection point is carried out isolation can by communication port every
From or Network Isolation platform realize, also will the data isolation communication of log information and other collection points of target collection point, also
I.e. while the collection of log information, i.e. achieve the isolation of preliminary log information, reduce the dry of unrelated log information
Disturb, improve accuracy and the analysis efficiency of user of daily record location.
Further, the log information of isolation collection point is combined to form information object set, log information is adopted
Collection is similar to pipeline thread, log information is read from each collection point isolated according to the acquisition parameter in default configuration file
Take out, according to the requirement of user, the log information of the collection point of isolation can be diverted in different JFSs,
Make in server the division of labor between each device definitely, and the log information of different acquisition point can be carried out by shunting
Classification, is polymerized the log information of same collection point, enables log information precisely to store, reduce unrelated log information
Interference, improve daily record location accuracy and the analysis efficiency of user.
It addition, information object set is converted to specified format according to the configuration file preset, and carry out with specified format
Storage, the tissue making log information is more flexible, and it is more convenient to process.
Log information is the information of hard disk, software and system problem in record system, simultaneously can be to occur in monitoring system
Event, such as include application log, security log, system journal and web administration daily record etc..
Such as, isolation collection point A, isolation collection point B and isolation collection point C, it is respectively used to the A service area of acquisition server
Fisrt fault log information, the second fault log information of B service area of server and server C service area the 3rd therefore
Barrier log information, according to acquisition time by Fisrt fault log information, the second fault log information and the 3rd fault log information
Being stored in application stack with key assignments form, and combine to be formed information object set, each information object set can will be adopted
The collection time period is as mark, and then can determine collection corresponding when breaking down when any of the above-described server fail
Time period, and the information object set of correspondence, improve accident analysis efficiency and the accuracy rate of log information, decrease computing
Amount.
In technique scheme, it is preferable that also include: preset unit 208, the collection for preset configuration file is joined
Number, wherein, acquisition parameter includes trigger condition, the log information collection that specified format, log information type, log information gather
At least one in the store path of period and information object set.
In this technical scheme, by the acquisition parameter of preset configuration file, it is achieved that the flexible storage of log information, with
Time ensure that the effectiveness that log information gathers, acquisition parameter includes what specified format, log information type, log information gathered
Trigger condition, log information gather at least one of the store path of period and information object set, by default acquisition parameter,
The log information of different acquisition point can be diverted in different JFSs.
Specifically, it is intended that form includes that W3C extension log file formats, Apache access log form, ArcSight are public
The form such as event format and IDMEF based on XML altogether, and different according to the application program performed, and log information type is the most not
With, such as determine according to test plan execution process logs, global positioning system (Global Positioning System, GPS)
Position information log and call information daily record etc., log information type can also be concrete according to debugging, warning, information, mistake etc.
Executive mode makes a distinction.Can indicate according to the fixing testing time for the log information collection period.
In any of the above-described technical scheme, it is preferable that also include: query unit 210, it is used for inquiring about isolation collection point raw
The log information become；Acquiring unit 212, for according to the configuration file preset, obtaining the daily record corresponding with log information type
Information；Module units 204 is additionally operable to: combine the log information obtained to form information object set.
In this technical scheme, the daily record corresponding with the log information type preset by inquiry log information acquisition is believed
Breath, with combination to form information object set, it is achieved that the polymerization to similar log information, enters the log information of same format
Row batch processing, improves the work efficiency of server.
Specifically, combining log information to form information object set according to default configuration file, configuration file is used
In describing the assembling form of log information, such as according to default configuration file by log information with corresponding application name
The forms such as title, log information size, log information acquisition time and log information content assemble.
In any of the above-described technical scheme, it is preferable that memory element 206 is additionally operable to: by information object set with key assignments
Form be stored in the application stack of server.
In this technical scheme, by information object set to be stored in the application stack of server with the form of key assignments
In, it is achieved that editor and the lookup to the log information in information object set.
Specifically, key assignments is positioned at registration table structural chain end, comprises the actual disposition information used when application program performs
And data, generally include several data type (such as character string, binary value, DWORD value etc.), to adapt to the use of varying environment
Demand, according to key assignments can be easily accomplished in the application stack to server storage the increase of data of log information, deletion,
The operation such as amendment and inquiry.
Wherein, the storage operation of key assignments (key-value in distributed data system) form includes:
(1) perform tree search and insert the position at data (i.e. the data of log information) place with location；
(2) father node of this position is locked；
(3) new leaf node is created；
(4) being write by leaf node, this write operation occurs in internal memory (application stack of the application), and returns one
Individual numerical value, this numerical value determines leaf node and writes the position of hard disk；
(5) amendment father node points to quoting of this leaf node, and this father node had both held the leaf node of sensing internal memory
Quote and hold the numerical value of position of disk；
(6) this father node of labelling is " dirty " (data meaning in internal memory do not appear in disk)；
(7) unlock this father node, by node inspection operation, the data of the log information of father node sensing are written to firmly
Dish.
Inquiry with key assignments form is operated and includes:
Navigate to store the leaf node of the structure tree of log information, inquired about the number reading log information by major key
According to.
In any of the above-described technical scheme, it is preferable that also include: updating block 214, for obtaining user to configuration
When the renewal of file instructs, the configuration file after updating replaces with default configuration file.
In this technical scheme, the configuration file after updating replaces with default configuration file, start by set date pipe line
Journey, all can read up-to-date configuration file when pipeline thread starts every time, thus the shunting gatherer process of Update log information, carry
Rise the motility that log information gathers.
Fig. 3 shows the schematic block diagram of server according to an embodiment of the invention.
As it is shown on figure 3, server 300 according to an embodiment of the invention, including described in any of the above-described technical scheme
Log Collect System 200, therefore, this server 300 has and the Log Collect System 200 described in any of the above-described technical scheme
Identical technique effect, does not repeats them here.
Fig. 4 shows the schematic flow diagram of log collection method according to another embodiment of the invention.
As shown in Figure 4, including: step 402, configuration log collection point information, and target collection point is isolated；Step
404, log parameters storage file is carried out parameter configuration, generates the configuration file preset；Step 406, isolation is adopted by server
The log object of the log information conversion specified format of collection point, and be stored in the application stack of server with key assignments form；Step
Rapid 408, according to default configuration file, from application stack, obtain log object, combine in chronological order to form information pair
As set；Step 410, according to default configuration file, by the store path of information object set, information object set unloading
In the JFS of server.
In this technical scheme, by being temporarily stored in application stack according to the collection point of isolation by log information, treat pipeline
After thread starts, log information is read out from each collection point isolated is gathered according to default configuration file, pipeline
The log information of collection point can be diverted in different JFSs by thread according to the requirement of user, and flexible organization is
Whole log content.
Owing to all can read up-to-date configuration when pipeline thread starts every time, it is possible to realize the change day of Real-time and Dynamic
The flow direction of will content, reaches the function of dynamic filtration and shunting.
Step in embodiment of the present invention method can carry out order according to actual needs and adjust, merges and delete.
One of ordinary skill in the art will appreciate that all or part of step in the various methods of above-described embodiment is can
Completing instructing relevant hardware by program, this program can be stored in a computer-readable recording medium, storage
Medium include read only memory (Read-Only Memory, ROM), random access memory (Random Access Memory,
RAM), programmable read only memory (Programmable Read-only Memory, PROM), erasable programmable is read-only deposits
Reservoir (Erasable Programmable Read Only Memory, EPROM), disposable programmable read only memory (One-
Time Programmable Read-Only Memory, OTPROM), the electronics formula of erasing can make carbon copies read only memory
(Electrically-Erasable Programmable Read-Only Memory, EEPROM), read-only optical disc (Compact
Disc Read-Only Memory, CD-ROM) or other disk storages, disk memory, magnetic tape storage or can
For carrying or store any other medium computer-readable of data.
Fig. 5 shows the schematic block diagram of Log Collect System according to another embodiment of the invention.
As it is shown in figure 5, Log Collect System 500 according to another embodiment of the invention, including: acquisition parameter configures
Unit 502, for generating default configuration file and collection point information, wherein, the configuration file preset includes: specified format,
When log information type (such as debugging, warning, information and mistake etc.), the trigger condition of log information collection, log information gather
The store path of section and described information object set is medium, and collection point information includes the identification information of collection point and retouching of collection point
State information；Log information module units 504, for gathering log information, and according to default from the isolation collection point of server
Configuration file, combines to be saved in the application stack of server by log information according to form after forming information object set,
To wait that pipeline thread is acquired；Daily record piping unit 506, for reading and organize final daily record to believe from application stack
Breath, daily record piping unit 506 needs to realize scheduling feature, one pipeline thread of start by set date, according to default configuration file, from
Application stack takes out in turn log information, and organizes together, call the service that log information memory element 508 provides, will
Be converted in the JFS that the information object set of specified format stores server；Log information memory element 508,
For being converted into the information object set of specified format, it is common that byte stream form, store server according to configuration file
JFS.
Unit in embodiment of the present invention terminal can merge according to actual needs, divides and delete.
Technical scheme is described in detail, it is contemplated that how correlation technique improves in daily record above in association with accompanying drawing
The technical problem of work efficiency when location and analysis fault, the present invention proposes a kind of new log collection scheme, by root
According to the identification information of collection point, collection point is isolated, according to default configuration file, the log information that isolation collection point generates is entered
Row assembles and is also converted to specified format, to store to the JFS of server, it is achieved that carry out log information point
Stream processes, and improves the work efficiency of server, adds motility and accuracy that log information gathers, decreases unrelated day
The interference of will information, improves the analysis efficiency to log information.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for the skill of this area
For art personnel, the present invention can have various modifications and variations.All within the spirit and principles in the present invention, that is made any repaiies
Change, equivalent, improvement etc., should be included within the scope of the present invention.
Claims (11)
1. a log collection method, it is characterised in that including:
Target collection point is isolated by the identification information of the collection point according to server, to form isolation collection point；
Combine according to the log information that described isolation collection point is generated by default configuration file to form information object set；
According to default configuration file, described information object set is converted to specified format, and by the information of described specified format
Object set stores to the JFS of described server.
Log collection method the most according to claim 1, it is characterised in that the mark letter in the collection point according to server
Before described collection point is isolated by breath, also include:
Presetting the acquisition parameter of described configuration file, wherein, described acquisition parameter includes described specified format, log information class
In the store path of trigger condition, log information collection period and described information object set that type, log information gather extremely
Few one.
Log collection method the most according to claim 2, it is characterised in that according to default configuration file by described isolation
The log information that collection point generates combines to form information object set, specifically includes following steps:
Inquire about the log information that described isolation collection point generates；
According to described default configuration file, obtain the log information corresponding with described log information type；
The log information obtained is combined to form described information object set.
Log collection method the most according to claim 3, it is characterised in that also include:
Described information object set is stored in the application stack of described server with the form of key assignments.
Log collection method the most according to any one of claim 1 to 4, it is characterised in that also include:
When obtaining user and instructing the renewal of configuration file, the configuration file after updating replaces with described default configuration literary composition
Part.
6. a Log Collect System, it is characterised in that including:
Isolated location, described collection point is isolated by the identification information for the collection point according to server；
Module units, for combining the log information that isolation collection point generates to form information pair according to the configuration file preset
As set；
Memory element, for being converted to specified format according to the configuration file preset by described information object set, and by described
The information object set of specified format stores to the JFS of described server.
Log Collect System the most according to claim 6, it is characterised in that also include:
Presetting unit, for presetting the acquisition parameter of described configuration file, wherein, described acquisition parameter includes described appointment lattice
Formula, log information type, the trigger condition of log information collection, log information gather depositing of period and described information object set
At least one in storage path.
Log Collect System the most according to claim 7, it is characterised in that also include:
Query unit, for inquiring about the log information that described isolation collection point generates；
Acquiring unit, for according to described default configuration file, obtains the log information corresponding with described log information type；
Described module units is additionally operable to: combine the log information obtained to form described information object set.
Log Collect System the most according to claim 8, it is characterised in that
Described memory element is additionally operable to: described information object set is stored in the application heap of described server with the form of key assignments
In stack.
10. according to the Log Collect System according to any one of claim 6 to 9, it is characterised in that also include:
Updating block, for when obtaining user and instructing the renewal of configuration file, the configuration file after updating replaces with institute
State default configuration file.
11. 1 kinds of servers, it is characterised in that including: the Log Collect System as according to any one of claim 6 to 10.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610623314.6A CN106254109A (en) | 2016-08-02 | 2016-08-02 | Log collection method, Log Collect System and server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610623314.6A CN106254109A (en) | 2016-08-02 | 2016-08-02 | Log collection method, Log Collect System and server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106254109A true CN106254109A (en) | 2016-12-21 |
Family
ID=57607056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610623314.6A Pending CN106254109A (en) | 2016-08-02 | 2016-08-02 | Log collection method, Log Collect System and server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106254109A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106777371A (en) * | 2017-01-23 | 2017-05-31 | 北京齐尔布莱特科技有限公司 | Result collection system and method |
| CN107566525A (en) * | 2017-10-12 | 2018-01-09 | 福建富士通信息软件有限公司 | A kind of cross-platform data based on Kaa services uploads, notice, collocation method |
| CN109460392A (en) * | 2018-10-10 | 2019-03-12 | 东软集团股份有限公司 | Journal file acquisition method, device, readable storage medium storing program for executing and electronic equipment |
| CN109905253A (en) * | 2017-12-07 | 2019-06-18 | 航天信息股份有限公司 | A kind of log information acquisition method and device |
| CN110710168A (en) * | 2017-04-04 | 2020-01-17 | Netapp股份有限公司 | Intelligent thread management across isolated network stacks |
| CN110969457A (en) * | 2018-09-29 | 2020-04-07 | 中国移动通信集团浙江有限公司 | Mobile application log collection method and system |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101149829A (en) * | 2006-09-22 | 2008-03-26 | 阿里巴巴公司 | Client data centralized processing method and system |
| CN101257399A (en) * | 2007-12-29 | 2008-09-03 | 中国移动通信集团四川有限公司 | Service system united safe platform |
| CN101534213A (en) * | 2009-04-09 | 2009-09-16 | 成都市华为赛门铁克科技有限公司 | Acquisition method of log and log server |
| CN103236949A (en) * | 2013-04-27 | 2013-08-07 | 北京搜狐新媒体信息技术有限公司 | Monitoring method, device and system for server cluster |
| CN103401698A (en) * | 2013-07-02 | 2013-11-20 | 北京奇虎科技有限公司 | Monitoring system used for alarming server status in server cluster operation |
-
2016
- 2016-08-02 CN CN201610623314.6A patent/CN106254109A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101149829A (en) * | 2006-09-22 | 2008-03-26 | 阿里巴巴公司 | Client data centralized processing method and system |
| CN101257399A (en) * | 2007-12-29 | 2008-09-03 | 中国移动通信集团四川有限公司 | Service system united safe platform |
| CN101534213A (en) * | 2009-04-09 | 2009-09-16 | 成都市华为赛门铁克科技有限公司 | Acquisition method of log and log server |
| CN103236949A (en) * | 2013-04-27 | 2013-08-07 | 北京搜狐新媒体信息技术有限公司 | Monitoring method, device and system for server cluster |
| CN103401698A (en) * | 2013-07-02 | 2013-11-20 | 北京奇虎科技有限公司 | Monitoring system used for alarming server status in server cluster operation |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106777371A (en) * | 2017-01-23 | 2017-05-31 | 北京齐尔布莱特科技有限公司 | Result collection system and method |
| CN106777371B (en) * | 2017-01-23 | 2019-12-06 | 北京齐尔布莱特科技有限公司 | Log collection system and method |
| CN110710168A (en) * | 2017-04-04 | 2020-01-17 | Netapp股份有限公司 | Intelligent thread management across isolated network stacks |
| CN110710168B (en) * | 2017-04-04 | 2021-12-17 | Netapp股份有限公司 | Intelligent thread management across isolated network stacks |
| CN107566525A (en) * | 2017-10-12 | 2018-01-09 | 福建富士通信息软件有限公司 | A kind of cross-platform data based on Kaa services uploads, notice, collocation method |
| CN109905253A (en) * | 2017-12-07 | 2019-06-18 | 航天信息股份有限公司 | A kind of log information acquisition method and device |
| CN109905253B (en) * | 2017-12-07 | 2022-05-17 | 航天信息股份有限公司 | Log information acquisition method and device |
| CN110969457A (en) * | 2018-09-29 | 2020-04-07 | 中国移动通信集团浙江有限公司 | Mobile application log collection method and system |
| CN109460392A (en) * | 2018-10-10 | 2019-03-12 | 东软集团股份有限公司 | Journal file acquisition method, device, readable storage medium storing program for executing and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106254109A (en) | Log collection method, Log Collect System and server | |
| US6347374B1 (en) | Event detection | |
| CN107818150A (en) | A kind of log audit method and device | |
| CN110245035A (en) | A kind of link trace method and device | |
| CN107301115A (en) | Application exception is monitored and restoration methods and equipment | |
| CN107451034A (en) | A kind of big data cluster log management apparatus, method and system | |
| CN107832196A (en) | A kind of monitoring device and monitoring method for real-time logs anomalous content | |
| US8086694B2 (en) | Network storage device collector | |
| US9123006B2 (en) | Techniques for parallel business intelligence evaluation and management | |
| CN106104495A (en) | Information processor and the method for supervision | |
| CN106126551A (en) | A kind of generation method of Hbase database access daily record, Apparatus and system | |
| WO2015148328A1 (en) | System and method for accelerating problem diagnosis in software/hardware deployments | |
| CN109460307B (en) | Micro-service calling tracking method and system based on log embedded point | |
| CN103944761A (en) | Android log information filter method and system | |
| CN1321509C (en) | Universal safety audit strategies customing method based on mapping table | |
| CN103870549B (en) | The method for cleaning and device of petroleum geology software data | |
| CN103248511B (en) | A kind of analysis methods, devices and systems of single-point service feature | |
| CN115274122A (en) | Health medical data management method, system, electronic device and storage medium | |
| KR101171551B1 (en) | Event history memory device, event history tracking device, event history memory method, computer-readable recording medium recording event history memory program and computer-readable recording medium recording data structure | |
| Ostrand et al. | A Tool for Mining Defect-Tracking Systems to Predict Fault-Prone Files. | |
| CN109816338A (en) | Enterprise's rewards and punishments processing method, device, computer equipment and storage medium | |
| CN109710671A (en) | Realize the method and its database firewall system of the drainage of database manipulation data | |
| CN111538720A (en) | Method and system for cleaning basic data in power industry | |
| CN108989086B (en) | Open vSwitch illegal port operation automatic discovery and tracing system in OpenStack platform | |
| CN116738449A (en) | DSMM-based data security management and control and operation system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20161221 |
|
| RJ01 | Rejection of invention patent application after publication |