CN107508718A - Log Collect System, log collection unit and log collection method - Google Patents
Log Collect System, log collection unit and log collection method Download PDFInfo
- Publication number
- CN107508718A CN107508718A CN201610417661.3A CN201610417661A CN107508718A CN 107508718 A CN107508718 A CN 107508718A CN 201610417661 A CN201610417661 A CN 201610417661A CN 107508718 A CN107508718 A CN 107508718A
- Authority
- CN
- China
- Prior art keywords
- acquisition target
- daily record
- log collection
- journal file
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Debugging And Monitoring (AREA)
Abstract
This programme provides a kind of Log Collect System, including:Acquisition layer equipment, including more than one log collection unit;Log collection unit is preconfigured to be associated with using main body described in each of which, and gathers daily record data using the demand of main body according to described, and the daily record data of collection is uploaded into accumulation layer equipment;The accumulation layer equipment, including more than one data storage cell;Data storage cell described in each of which is corresponding with the log collection unit, and the daily record data uploaded for storing corresponding log collection unit, and the daily record data of storage is supplied into the use main body associated by the log collection unit.This programme provides a kind of log collection unit and a kind of log collection method simultaneously.This programme can solve the problems, such as that the distributed information log acquisition system of prior art lacks collection flexibility, system cost is too high and system deployment is excessively complicated.
Description
Technical field
This programme is related to daily record data processing technology field, and in particular to Log Collect System.This programme further relates to daily record
Collecting unit and log collection method.
Background technology
With the rapid development of cloud computing, largely occur one after another for the exclusive cloud system of specialized business scenario, this its
In outstanding person include PaaS (platform i.e. service) and SaaS (software is service) etc..These cloud systems are typically all using main body
Corresponding cloud service is provided on the premise of mandate by privately owned cluster;But due to using for main body, these cloud systems
Privately owned cluster be a black box, i.e. the bottom-up information (such as virtual machine and storage) of cloud service is using main body is invisible and nothing
Method control.In this case, the cluster running status of cloud system and the business procedure of related bottom-up information can be reflected
Daily record is vital;Cloud system is needed to provide the accurately daily record data with real-time business procedure using main body, to assist
Whether it checks cloud service safe and reliable.
At present, above-mentioned daily record number can be met by proposing several distributed massive logs acquisition system in the prior art
According to the demand of collection, still, for system architecture, the distributed Log Collect System of these prior arts is all quite multiple
It is miscellaneous, it is generally divided into three layers:
Include the log collection layer of multiple log collection units;
Include the data gathering layer of multiple data collection servers;
Include the data storage layer of multiple data storage servers.
Each log collection unit wherein in log collection layer is all according to system when gathering specific daily record data
Preset directly to carry out, it is impossible to make selection and the tune of collection according to the final real needs using main body of daily record data
It is whole;The acquisition mode of this fixation often results in the waste of collection lack of targeted and gathered data, while also requirement must be set
Put data gathering layer, supporting corresponding data collection server, come sort daily record data that log collection unit gathered and
Determine gathered daily record data relative to the ownership using main body.And completed in the sorting and determination operation of data gathering layer
Afterwards, then with data gathering layer by the data storage of collection to data storage layer.
From cost, the distributed acquisition system of prior art due to being divided into three structure sheafs in the vertical, it is necessary to
Data collection server is disposed to sort daily record data and determine the ownership of daily record data gathered;In the horizontal can not be in separate unit
Disposed on server, it is necessary to safeguard data collection server cluster and data storage server cluster.Such prior art
The lower deployment cost of distributed information log acquisition system is at a relatively high.
In actual application, the daily record data amount of many cloud systems using subject demandses is simultaneously little, also will not be right
Daily record data carries out the analysis and excavation of depth, normally only the specific complementary and daily record data of Process Character of demand.For example, make
When handling cloud service with big data, user performs some operations specific, it is necessary to check that operations specific performs in time in cloud system
Output journal, to ensure that when operations specific malfunctions error reason can be viewed.Use main body to these daily record datas
For, the distributed information log acquisition system of prior art lacks collection flexibility, and the cost of a full set of system is too high, and Account Dept
Administration is also excessively complicated.
The content of the invention
This programme provides a kind of Log Collect System, a kind of log collection unit and a kind of log collection method, with solution
Certainly the distributed information log acquisition system of prior art lacks collection flexibility, system cost is too high and system deployment is excessively complicated
The problem of.
This programme provides a kind of Log Collect System, including:
Acquisition layer equipment, including more than one log collection unit;Wherein
Each described log collection unit is preconfigured to be associated with using main body, and uses main body according to described
Demand and gather daily record data, and the daily record data of collection is uploaded to accumulation layer equipment;
The accumulation layer equipment, including more than one data storage cell;Wherein
Each described data storage cell is corresponding with the log collection unit, and for storing corresponding day
The daily record data that will collecting unit uploads, and the daily record data of storage is supplied to associated by the log collection unit made
Use main body.
Optionally, the Log Collect System, is deployed on cloud system.
Optionally, the daily record data, including:The daily record data of the business service process of cloud system.
Optionally, the acquisition layer equipment, in addition to:More than one monitoring unit;Wherein
Each described monitoring unit is corresponding with the log collection unit, and is touched by the log collection unit
Hair, and be used for:Monitor whether the log collection unit has been triggered;
If it is not, then trigger the log collection unit.
Optionally, the log collection unit is additionally operable to:
Monitor whether the monitoring unit has been triggered;
If it is not, then trigger the monitoring unit.
Optionally, the data storage cell, is additionally operable to:
By using cloud storage service, the daily record data of upload is stored under the account item using main body.
Optionally, the Log Collect System, it is deployed on the separate unit service server of the cloud system.
Optionally, more than two Log Collect Systems, while it is deployed in the same business clothes of the cloud system
It is engaged on device.
This programme also provides a kind of log collection unit, be preconfigured to be it is associated with using main body, including:Daily record is adopted
Collect component, and daily record transmitting assembly;
The log collection component, for according to the demand using main body, for the acquisition target of setting, gathering institute
State the daily record data of acquisition target;
The daily record transmitting assembly, for uploading gathered daily record data.
Optionally, in addition to:Dynamic configuration component, and dynamic monitor component;
The dynamic configuration component, for using journal file as acquisition target, and the acquisition target is added and gathered
List object, and the time interval of daily record monitoring and the time interval of log collection are set;
The dynamic monitor component, for the time interval monitored according to the daily record, acquisition target described in regular monitoring
List, and the acquisition target for having completed collection is removed from the acquisition target list.
Optionally, the daily record transmitting assembly, is additionally operable to:
Limit the size that maximum single uploads data.
Optionally, the daily record transmitting assembly, is additionally operable to:
Upload after completing, updated in end position information list is uploaded belonging to uploaded daily record data each time
Acquisition target upload end position;
The wherein described end position that uploads represents the last daily record data for completing to upload in corresponding journal file
End of data position.
Optionally, the daily record transmitting assembly, is additionally operable to:
By the upload end position information list of renewal, persistent storage medium is stored in.
Optionally, the log collection component, including:
At least one log collection sub-component, wherein
Each described log collection sub-component is corresponding with the acquisition target in the acquisition target list, for root
According to the demand using main body, whether changed according to the time interval and the acquisition target of the log collection, it is fixed
Phase gathers daily record data.
Optionally, it is additionally operable to:
According to the number change of acquisition target in the acquisition target list, dynamically increase and decrease the log collection sub-component.
Optionally, the acquisition target for having completed collection includes:
Journal file, the recent renewal time of the journal file exceed given threshold, and the day away from preset time point
The data renewal end position of will file is no more than the upload end position of the journal file;
The data renewal end position represents the daily record data of the last data renewal of the journal file in institute
State the end of data position in journal file.
Optionally, the dynamic configuration component, is additionally operable to:
Journal file set is added into the acquisition target list, using the journal file in the journal file set as
Acquisition target, and the time interval of the journal file set poll is set.
Optionally, the dynamic monitor component, is additionally operable to:
According to the time interval of the journal file set poll, the daily record text in acquisition target list described in periodic polling
Part set;
If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and
The acquisition target is added in the acquisition target list;
If the journal file in the journal file set is deleted, removed from the acquisition target list with being deleted
Except the corresponding acquisition target of journal file.
Optionally, the log collection sub-component, is used for:Acquisition target is new in acquisition target list described in taken at regular intervals
Increase daily record data.
Optionally, the log collection sub-component, is used for:
If the data renewal end position of the journal file exceedes the upload end position of the journal file, to institute
State the daily record data that newly increases uploaded between end position and data renewal end position and carry out taken at regular intervals.
This programme also provides a kind of log collection method, and main body is used corresponding to one, including:
According to the demand using main body, for the acquisition target of setting, the daily record data of the acquisition target is gathered;
Upload gathered daily record data.
Optionally, it is described according to the demand using main body in execution, for the acquisition target of setting, adopted described in collection
Before the step of collecting the daily record data of object, following step is performed:
Using journal file as acquisition target, and the acquisition target is added into acquisition target list, and daily record is set
The time interval of monitoring and the time interval of log collection;
The time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from the acquisition target
The acquisition target for having completed collection is removed in list.
Optionally, it is described to upload gathered daily record data;Including:
Limit the size that maximum single uploads data.
Optionally, it is described to upload gathered daily record data;Including:
Upload after completing, updated in end position information list is uploaded belonging to uploaded daily record data each time
Acquisition target upload end position;
The wherein described end position that uploads represents the last daily record data for completing to upload in corresponding journal file
End of data position.
Optionally, after completion is uploaded each time described in performing, for upper in end position information list is uploaded
After the step of acquisition target belonging to the daily record data of biography carries out uploading the renewal of end position information, following step is performed:
The upload end position information list of renewal is stored on persistent storage medium.
Optionally, it is described according to the demand using main body, for the acquisition target of setting, gather the acquisition target
Daily record data;Including:
According to the demand using main body, for the acquisition target in the acquisition target list, according to the daily record
Whether the time interval of collection and the acquisition target change, taken at regular intervals daily record data.
Optionally, the acquisition target for having completed collection includes:
Journal file, the recent renewal time of the journal file exceed given threshold, and the day away from preset time point
The data renewal end position of will file is no more than the upload end position of the journal file;
The data renewal end position represents the daily record data of the last data renewal of the journal file in institute
State the end of data position in journal file.
Optionally, the setting journal file adds acquisition target list as acquisition target, and by the acquisition target,
And the time interval of daily record monitoring and the time interval of log collection are set;Also include:
Journal file set is added into the acquisition target list, using the journal file in the journal file set as
Acquisition target, and the time interval of the journal file set poll is set.
Optionally, the time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from institute
State the acquisition target for being removed in acquisition target list and having completed collection;Also include:
According to the time interval of the journal file set poll, the daily record text in acquisition target list described in periodic polling
Part set;
If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and
The acquisition target is added in the acquisition target list;
If the journal file in the journal file set is deleted, removed from the acquisition target list with being deleted
Except the corresponding acquisition target of journal file.
Optionally, it is described whether to be changed according to the time interval and the acquisition target of the log collection, periodically
Gather daily record data;Including:
According to the time interval of the log collection, acquisition target newly increases in acquisition target list described in taken at regular intervals
Daily record data.
Optionally, acquisition target newly increases daily record data in acquisition target list described in the taken at regular intervals, including:
If the data renewal end position of the journal file exceedes the upload end position of the journal file, to institute
State the daily record data that newly increases uploaded between end position and data renewal end position and carry out taken at regular intervals.
This programme also provides a kind of computing device, be preconfigured to be it is associated with using main body, including:Processor and deposit
Reservoir;
The processor, for according to the demand using main body, for the acquisition target of setting, gathering the collection
The daily record data of object;Upload gathered daily record data.
Compared with prior art, this programme has advantages below:
This programme, can be corresponding directly against being carried out using main body by the way that log collection unit is associated with using main body
Daily record data collection, it is not necessary to the participation of data collection server, in vertical structure only have more simple daily record adopt
Two layers of collection and data storage;Meanwhile this programme can use the supporting data storage cell of a log collection unit
Form a complete Log Collect System, it is not necessary to which too fat to move distributed type assemblies, Log Collect System can be disposed easily
Used in enterprising exercise of separate unit service server.
This programme provide Log Collect System can according to using main body particular demands, flexible configuration acquisition target,
Implement targetedly log data acquisition, improve the efficiency of log data acquisition, it also avoid collection redundant data and cause
Waste.
The Log Collect System that this programme provides, due to simple in construction, only it can be disposed on separate unit service server,
It can be disposed on any number of server, autgmentability is relatively good;Meanwhile the Log Collect System of this programme need not service
The support of device cluster, the cost of system deployment is than relatively low.
The daily record data that the Log Collect System of this programme is gathered, both for specific using main body, these daily records
Data will not mutually mix with other any daily record datas using main body, while these daily record datas are also stored in for specific
Using the memory space of main body, the security and privacy of the data of Log Collect System be all higher.
The Log Collect System of this programme can also carry out storing daily record data using independent cloud storage service, so as to enter one
Step has simplified the structure of Log Collect System, reduces the cost of system deployment, also ensure that the security of daily record data and hidden
Private.
Brief description of the drawings
Fig. 1 is a kind of structural representation of the distributed information log acquisition system for prior art that this programme provides;
Fig. 2 is a kind of structural representation for Log Collect System that this programme first embodiment provides;
Fig. 3 is the signal that a kind of log collection unit that this programme first embodiment provides mutually monitors with monitoring unit
Figure;
Fig. 4 is a kind of structural representation for log collection unit that this programme second embodiment provides;
Fig. 5 is a kind of schematic diagram for upload end position information list that this programme second embodiment provides;
Fig. 6 is a kind of flow chart for log collection method that this programme 3rd embodiment provides;
Fig. 7 is a kind of flow chart for log collection method preferred steps that this programme 3rd embodiment provides;
Fig. 8 is a kind of schematic diagram for computing device that this programme fourth embodiment provides.
Embodiment
Many details are elaborated in the following description in order to fully understand this programme.But this programme can be with
Much it is different from other manner described here to implement, those skilled in the art can be in the situation without prejudice to this programme intension
Under do similar popularization, therefore this programme is not limited by following public specific implementation.
In this programme, a kind of Log Collect System, a kind of log collection unit, a kind of log collection side are each provided
Method and a kind of computing device.Below in conjunction with the accompanying drawings, it is described in detail one by one in the following embodiments.
Fig. 2 is refer to, it is a kind of structural representation for Log Collect System that this programme first embodiment provides.It is described
Log Collect System, including:
Acquisition layer equipment 101, including more than one log collection unit 101-1;Wherein, each described log collection
Unit 101-1 is preconfigured to be associated with using main body, and gathers daily record data using the demand of main body according to described,
And the daily record data of collection is uploaded to accumulation layer equipment 103;
The accumulation layer equipment 103, including more than one data storage cell 103-1;Wherein, each described data
Memory cell 103-1 is corresponding with the log collection unit 101-1, and for storing corresponding log collection unit
The daily record data that 101-1 is uploaded, and the daily record data of storage is supplied to associated by the log collection unit 101-1 made
Use main body.
The Log Collect System of this programme, include the acquisition layer equipment 101 of multiple log collection units and there are more numbers
According to the accumulation layer equipment 103 of memory cell;Wherein, each log collection unit is by the use with particular log data in advance
Main body is associated, and gathers using the demand of the specific usage log data of main body according to this daily record data and upload and adopt
The daily record data of collection to accumulation layer equipment 103 data storage cell;Each data storage cell both corresponds to particular log
Collecting unit, and the daily record data gathered dedicated for particular log collecting unit corresponding to storage, and the day by storage
Will data are supplied to the use main body associated by this log collection unit.
In this programme is exactly the actually user of daily record data, including logical outside Log Collect System using main body
User that the Log Collect System often said is serviced and other need to use the equipment or facility of daily record data.In this programme
In, each log collection unit, which associates, specifically uses main body;Log collection unit is being configured to associate specific use master
It is as specific to use the exclusive use of main body after body, it is impossible to be configured association again and use main body to other.For example, in this implementation
In example, as shown in Fig. 2 each corresponding use respectively of log collection unit 1, log collection unit 2 and log collection unit 3
Main body, i.e. daily record collecting unit 1 are corresponding to use main body 1, and each corresponding use is led for log collection unit 2 and log collection unit 3
Body 2, and use main body 2 then to correspond to two log collection units.
Log collection unit in this programme, for the collection and upload of daily record data to be embodied.The of this programme
Log collection unit will will be described in detail in two embodiments.
Daily record data in this programme, can be the daily record data of the business service process of cloud system, i.e. cloud system is carrying
During business service of the supply using main body cloud system, about the business procedure such as specifically being calculated, being handled and being stored
Daily record data;Daily record data or use other required any type of data of main body in this programme, such as industry
Some specific service state of business server etc..Daily record data in this programme is also not excluded for applying and is deployed in addition to cloud system
Situation, for example, it is also possible to be the daily record data on separate unit service server.
Log Collect System in this programme, can be the Log Collect System being deployed on cloud system.It can also be portion
Administration is applied to other distributed and non-distributed network system even single server systems in addition to cloud system.In this reality
Apply in example, for the purpose of simplifying the description, the Log Collect System being preferably deployed on cloud system.Generally, in this programme
Also it is the use main body of cloud system simultaneously using main body, the business processing service of cloud system offer is provided.
It please also refer to Fig. 1, be in place of the difference of the distributed information log acquisition system of this programme and prior art:
First, each log collection unit of real data collection, quilt are carried out in the collecting device layer 101 of this programme
Be configured to using main body it is associated (in one embodiment, a log collection unit can be associated with one and use main body,
In yet another alternative embodiment, a log collection unit can be associated with it is multiple use main body, replaced at also one real
Apply in example, multiple log collection units can be associated with one and use main body);And the daily record of the Log Collect System of prior art
Collecting unit does not establish corresponding relation with any one using main body.One of skill in the art will understand that at least it is based on
For each log collection unit of this programme with the clear and definite incidence relation established using main body, log collection unit can
With the real needs using main body to daily record data associated according to it, flexible configuration acquisition target, so as to implement targetedly
Log data acquisition, improve collecting efficiency, avoid gathering unwanted data;And the day of the Log Collect System of prior art
The log data acquisition of will collecting unit is that system is preset and changeless, it is impossible to according to the specific of specifically used main body
Demand carrys out the log data acquisition of flexible customization log collection unit;
Second, because each log collection unit of this programme using main body with establishing clear and definite incidence relation,
Then further log collection unit can establish corresponding relation with this using the data storage cell of main body, and can directly by
The data gathered are uploaded to corresponding data storage cell by log collection unit, thus, the Log Collect System of this programme
The prerequisite data gathering layer of institute in the prior art is eliminated in vertical structure, greatly simplify the structure of whole system,
Realize light-weighted system deployment effect.By contrast, log collection unit of the prior art does not make with any one
Corresponding relation is established with main body (and the data storage cell of main body is used for this), it is therefore necessary to disposes data gathering layer
The daily record data and confirm which gathered attribution data use main body in that sorting log collection unit is gathered, so make
Obtaining the structure of whole system tends to be complicated, improves cost of implementation;
3rd, because each log collection unit of this programme using main body and/or data storage cell with being established
Association or corresponding relation, then the daily record data that log collection unit is gathered be specific to it is specific use main body, and adopted by daily record
Collection unit is directly uploaded in the specific data storage cell using main body, thus, in daily record data from collecting storage
In whole process, the specific daily record data using main body will not mutually be mixed with other using the daily record data of main body, such day
The security and privacy of will data can be guaranteed;And in the Log Collect System of prior art, different daily records
Data (may belong to different use main bodys) need to be divided via data gathering layer in collection, collection and storing process
Pick and confirm, therefore be mingled in and together, be stolen which increase daily record data, replicated, the possibility of malice forwarding, be unfavorable for
Data-privacy is protected.
It should be noted that because each log collection unit of this programme is with using main body and/or data storage cell
Establish association or corresponding relation, the flowing of the daily record data of collection in the Log Collect System of this programme can only be longitudinal direction not
It can be horizontal, i.e., data storage cell can only be uploaded to from log collection unit, then use is supplied to by data storage cell
Main body.For example, the daily record data gathered from log collection unit 2, is only possible to upload to data storage cell 2, then provides
Give use main body 2.This daily record data can not possibly be supplied to data storage cell 3.
As shown in figure 3, the acquisition layer equipment 101 in this programme, in addition to it can include multiple log collection units, also
More than one monitoring unit 101-3 can be included;Wherein, each described monitoring unit 101-3 with the log collection
Unit 101-1 is corresponding, and is triggered by the log collection unit 101-1.How many Log Collect System can is corresponding
Supporting how many individual monitoring units.
Monitoring unit is mainly used in monitoring whether log collection unit 101-1 has been triggered;If log collection unit
It is not triggered, then triggers log collection unit 101-1.Moreover, log collection unit 101-1 can also be monitored in turn
This monitoring unit 101-3, for monitoring whether this monitoring unit 101-3 has been triggered;If monitoring unit 101-3 does not have
It is triggered, then triggers monitoring unit 101-3.
Monitoring and log collection unit monitoring to monitoring unit of the monitoring unit to log collection unit, can be monitoring
Monitoring in real time or at regular intervals interval carry out periodicity monitoring once;Judge log collection unit or
The standard that monitoring unit is not triggered can be determined flexibly.For example, monitoring unit can at set intervals, periodically
1 monitoring to log collection unit was carried out every 1 minute;The mark such as whether can be acquired by log collection unit
Standard judges whether log collection unit is triggered.
The data storage cell 103-1 in accumulation layer equipment 103 in this programme, corresponding to a specific log collection
Unit, that is, specifically use main body corresponding to one.Data storage cell can use a variety of storages in the specific implementation
Mode and storage medium, to store the daily record data that corresponding log collection unit is gathered, for example, directly in single server
On hard disk on store, either storage network hard disc in a local network or storage is what Log Collect System was disposed
Under the cloud storage account item that same cloud system provides, etc..Allow for obtaining day for being stored of data storage cell using main body
Will data, either using which kind of concrete mode, such as directly read from home server hard disk or this LAN network hard disc,
Or downloaded from network etc..
The data storage cell 103-1 in accumulation layer equipment 103 in this programme, is additionally operable to take by using cloud storage
Business, the daily record data of upload is stored under the account item using main body.
When carrying out data storage using the cloud storage service for providing separate storage account, main body can also be used by one
Daily record data is all stored in ownership, and this is used under the cloud storage account item of main body.Now, cloud storage can on the one hand be utilized
Independence and privacy between system account, ensure the safety of the daily record data of storage;On the other hand daily record can also be utilized
Cloud storage service outside acquisition system, further simplify the structure of Log Collect System;This is substantially equivalent to by daily record
Contracted out outside the accumulation layer equipment of acquisition system, implement the work(of the accumulation layer equipment of Log Collect System using cloud storage service
Can, the whole Log Collect System of lightweight.Certainly, in one embodiment, can be by a daily record data using main body
It is stored in and belongs to this one or more cloud storage account using a cloud storage system of main body;It is real in another replacement
Apply in example, a daily record data using main body can also be stored in multiple accounts of multiple cloud storage systems.The present embodiment
It is preferred that the cloud storage service provided in itself by the cloud system of deployment by Log Collect System is provided, the efficiency of such data storage
Compare high.For example, in the present embodiment, possess independent cloud in the cloud system that Log Collect System is disposed using main body 1
Authority is stored, corresponding data storage cell can is using master by using the cloud storage account and password using main body 1
The daily record data that corresponding log collection unit uploads is stored under the cloud storage account item of body 1.
Log Collect System in this programme, horizontal extension can be carried out.Can be according to the specific of log data acquisition
Need, corresponding one or more multiple Log Collect Systems using main body are deployed in multiple service servers of cloud system
On, it is this extending transversely to be virtually limitless;Log Collect System in this programme, horizontal simplify can also be carried out.
Can be a daily record by a corresponding minimum Log Collect System using main body according to the specific needs of log data acquisition
Collecting unit and a data storage cell are deployed directly on the separate unit service server of cloud system;Or corresponding two are made
With more than two log collection units of main body, it is deployed on same service server of cloud system.For example, in this implementation
In example, the Log Collect System in Fig. 2 is to be deployed on separate unit service server, wherein log collection unit 1 and daily record
Collecting unit 2,3 is respectively using main body 1 and using main 2 collection daily record data.
Fig. 4 is refer to, it is a kind of structural representation for log collection unit that this programme second embodiment provides.We
The log collection unit 101 of case, including:Log collection component 201, and daily record transmitting assembly 203;
The log collection component 201, for according to the demand using main body, for the acquisition target of setting, adopting
Collect the daily record data of the acquisition target.
Log collection component 201, the use according to corresponding to log collection unit to be used among log collection unit 101
Main body, for these journal files as acquisition target, implements the collection of specific daily record data to the demand of journal file.
The daily record transmitting assembly 203, for uploading gathered daily record data.
Daily record transmitting assembly 203, it is used to upload what log collection component 201 was gathered among log collection unit 101
Daily record data.
It can be seen that a minimum log collection unit, can include a log collection component and a daily record upload group
Part;If add a data storage cell, it is possible to form a minimum Log Collect System.
The log collection unit of this programme, in addition to:Dynamic configuration component 205, and dynamic monitor component 207.
The dynamic configuration component 205, for using journal file as acquisition target, and the acquisition target is added and adopted
Collect list object, and the time interval of daily record monitoring and the time interval of log collection are set.
Dynamic configuration component 205 among the log collection unit of this programme, for using journal file as acquisition target,
And this acquisition target is added into acquisition target list, i.e., newly-installed acquisition target is added into acquisition target list, Yi Jishe
Put the time interval of daily record monitoring and the time interval of log collection is set.Progress daily record can be set to adopt by this component 205
Basic parameter needed for collection, and monitor and manage collection activity by the basic parameter of these settings.
The acquisition target of this programme can be diversified forms, can be the daily record text of the business service process of cloud system
During the business service using main body cloud system is supplied to, record is relevant specifically to be calculated, is handled for part, i.e. cloud system
With storage etc. business procedure journal file;Acquisition target in this programme or using main body it is required other
The data for type of anticipating, such as some specific service state of service server etc..Acquisition target in this programme is also not excluded for should
With the situation being deployed in addition to cloud system, for example, it is also possible to be the journal file on separate unit service server.In the present embodiment
In, it is specific journal file corresponding to acquisition target to simplify description, i.e., each acquisition target is exactly to add collection pair
As a journal file in list;It is being to confirm that this journal file is shape to be collected using journal file as acquisition target
State, do not make any change or mark in itself to journal file, journal file is simply included in classification to be collected.Specific
During implementation, file name asterisk wildcard can be set, to support to meet that the journal file of this title asterisk wildcard is used as acquisition target.
Acquisition target list contains the acquisition target that log collection unit is acquired, that is, contains state to be collected
Journal file, daily record monitor component determine its specific acquisition target monitored according to the content of acquisition target list, and daily record is adopted
Collection component similarly determines its specific journal file gathered according to acquisition target list.For example, 20 journal files are made
Acquisition target list is added for acquisition target, that is, it is state to be collected to confirm this 20 journal files.
The time interval of daily record monitoring and the time interval of log collection, are all the automatic or manual tools set manually of system
The period of the specific duration of body, typically in seconds.The two time intervals are all for follow-up daily record monitoring step and day
The step of will gathers does parameter preparation;Time interval it is shorter, daily record monitoring and log collection are more real-time.For example, set
The time interval of daily record monitoring is 30 seconds, and the time interval for setting log collection is 10 seconds.
The dynamic monitor component 207, for the time interval monitored according to the daily record, collection pair described in regular monitoring
As list, and the acquisition target for having completed collection is removed from the acquisition target list.
Dynamic monitor component 207 is used for the time interval monitored according to the daily record set by dynamic configuration component 205, periodically
The acquisition target list set by dynamic configuration component 205 is monitored, and is removed from this acquisition target list and has completed to adopt
The acquisition target of collection;Dynamic monitor component 207 is additionally operable to specific monitoring collection list object, and removes and complete adopting for collection
Collect object, to prevent the situation that acquisition target is more and more, swarms in acquisition target list.
Dynamic monitor component 207 is the time interval for being monitored according to the daily record set by dynamic configuration component 205, fixed
Phase triggering, i.e., the time interval monitored every the daily record of one section of setting, once dynamic monitor component 207 is triggered, and so on.
Adopted in the design parameter needed for the setting log collection of dynamic configuration component 205 and the monitoring of dynamic monitor component 207
After collecting list object, the log collection component 201 of this programme can be further refined.Log collection component 201, if in addition to
A dry log collection sub-component 201-1, wherein, each described log collection sub-component with the acquisition target list
One acquisition target is corresponding, for according to the demand using main body, time interval and institute according to the log collection
State whether acquisition target changes, taken at regular intervals daily record data.
Log collection component 201 includes one or more log collection sub-component 201-1 for corresponding to single acquisition target;
Wherein, each described log collection sub-component mutually corresponds with the acquisition target in the acquisition target list, adopts
Collect that how many acquisition target in list object just corresponds to the how many individual log collection sub-components of triggering;Each log collection subgroup
Part is all used for using the real needs of main body corresponding to the log collection unit belonging to, according to the time of the log collection of setting
Whether interval and acquisition target change in file level, the corresponding daily record data of taken at regular intervals.Such as in the present embodiment,
As shown in figure 3, log collection component includes 3 log collection sub-components, they correspond to 3 daily records as acquisition target respectively
File, carry out the collection of daily record data.
Log collection component 201 also includes:According to the number change of acquisition target in the acquisition target list, dynamic increases
Subtract the log collection sub-component.If the quantity increase of acquisition target, log collection component dynamic in acquisition target list
The quantity of log collection sub-component is adjusted, it is corresponding to increase log collection sub-component, make increased log collection sub-component and increase
Acquisition target it is corresponding;If the quantity of acquisition target is reduced in acquisition target list, log collection component dynamic adjusts
The quantity of log collection sub-component, reduces corresponding log collection sub-component, and corresponding remove corresponds to by from acquisition target list
The log collection sub-component of the acquisition target of removal.The quantity of log collection sub-component is with collection pair in acquisition target list
The number change of elephant and dynamic change.
The daily record transmitting assembly 203 of this programme, is additionally operable to:Limit the size that maximum single uploads data.
Daily record transmitting assembly can follow a upload when for uploading daily record data that log collection component is gathered
The limitation of bandwidth, i.e. maximum single upload size of data without departing from the restriction bandwidth upper limit.Set uploading bandwidth limitation be in order to
Prevent the network traffics of service server that Log Collect System causes to be disposed from blocking, ensure the stability of system operation.
For example, if limiting maximum single uploads the size of data as 500KB, it is assumed that corresponds to some and uses main body, daily record
Collecting unit is directed to some acquisition target, the new daily record data for gathering 600KB, then when triggering daily record transmitting assembly, Zhi Nengyi
Secondary upload 500KB daily record data.
Daily record transmitting assembly 203, is additionally operable to:Upload each time after completing, in end position information list is uploaded
The upload end position of acquisition target belonging to the uploaded daily record data of renewal;Wherein described upload end position represents nearest
End of data position of the daily record data for once completing to upload in corresponding journal file.
Daily record transmitting assembly 203, can also be each when for uploading daily record data that log collection component is gathered
After secondary upload is completed, in end position information list is uploaded, enter for the acquisition target belonging to the daily record data of upload
Row uploads the renewal of end position.
The upload end position of acquisition target, it is to be directed to particular acquisition object, the last time completes upload daily record data and existed
As the Data Position in the journal file of particular acquisition object, this upload end position can determine to have completed to upload and
Position boundary of the daily record data not yet uploaded in journal file.For example, the log file size of an acquisition target
For 1000KB.In one embodiment, the last data for completing to upload daily record data are determined by uploading end position
End position is at 800KB, then can determine in this journal file, upload 800KB daily record datas, is also had
1000KB-800KB=200KB daily record datas not yet upload.
End position information list is uploaded, is the summary table for the upload end position information for listing acquisition target, it specify that
The last end position for uploading data in journal file of each acquisition target.Upload end position information list tool
Body form is exactly one-to-one two-dimensional table, refer to Fig. 5, and it is that the upload that this programme second embodiment provides terminates
The content schematic diagram of positional information list.It is the daily record text as acquisition target to upload the row in end position information list
Part, another row are that the last of the corresponding journal file of record uploads the particular location terminated.
Because the other assemblies outside the daily record uploading unit of this programme need to obtain or update the upload knot of acquisition target
Beam position information, after triggering daily record transmitting assembly each time and completing to upload daily record data, daily record transmitting assembly can also be used
In the upload knot that corresponding acquisition target to the acquisition target belonging to daily record data, is carried out in end position information list is uploaded
The renewal of beam position information.
For example, in the present embodiment, a log file size as acquisition target is 1000KB, the last time is namely
The end position that the last time completes to upload daily record data is 800KB, i.e., in end position information list is uploaded, record uploads
End position is 800KB.Continue to acquire 200KB daily record datas again since 800KB positions afterwards, correspondingly upload again
150KB daily record datas among 200KB, then it is 1000KB for this file size in end position information list is uploaded
Acquisition target, it is 950KB to update it to upload end position.
Completed in triggering daily record transmitting assembly after renewal uploads end position information list, daily record transmitting assembly can be with
For by the upload end position information list of renewal, being stored in persistent storage medium.
By carrying out persistent storage to the upload end position information list of renewal, for example, in the correlation of local hard drive
Storage catalogue, it can prevent that end position information is uploaded caused by service server powers off the machine of delaying suddenly loses, so as to avoid
Repeat to upload daily record data, and improve the stability and efficiency of Log Collect System.
The acquisition target for having completed collection includes:Journal file, during the recent renewal of the journal file
Spacing preset time point exceedes given threshold, and the data renewal end position of the journal file is no more than the journal file
Upload end position;The data renewal end position represents the daily record number of the last data renewal of the journal file
According to the end of data position in the journal file.
The update mode of journal file involved by this programme can be diversified forms, for example, it may be incremental update,
Can also be covering renewal or other forms.If journal file is by the way of covering updates, without the concern for daily record
The data renewal end position of file, directly uploads journal file when uploading daily record data, it is not required that be considered as adopting
Whether the journal file of collection object has been completed to gather, and is directly viewable whether journal file was once uploaded and the last data
The time of renewal.In the present embodiment, for the purpose of simplifying the description, employ setting journal file using incremental update and every time
The mode of new data is all added during renewal in end of file.
If as the journal file of an acquisition target in acquisition target list, last update time interval is preset
Time point (such as when setting preset time point as specific monitoring collection object) exceedes given threshold, and journal file is most
Nearly data renewal end position is no more than the upload end position of the journal file, then can make this acquisition target
To have completed the acquisition target of collection.The recent renewal time of acquisition target can be from corresponding journal file attribute directly
Obtain;Preset time point is exactly typically when being specifically monitored, naturally it is also possible to sets other rational time points;Set threshold
Value depends on concrete condition, can specifically be set, for example, 1 day or 3 days.Recent renewal time exceedes away from preset time point
Given threshold is the time limit that the recent renewal time of journal file has exceeded setting, i.e. renewal time has exceeded setting
Time.The last end position information that uploads of acquisition target can obtain from uploading in end position information list;Data
Renewal end position is no more than the last end position that uploads and shown, on this acquisition target (journal file), daily record number
Finished according to all having uploaded.
Dynamic monitor component 207, for the time interval monitored according to the daily record, acquisition target described in regular monitoring arranges
Table, and the acquisition target for having completed collection is removed from the acquisition target list, as periodically acquisition target is arranged
Acquisition target in table is monitored, if specific acquisition target meets above-mentioned time conditions and data qualification simultaneously, so that it may
To determine that this acquisition target to have completed to gather, when being updated to acquisition target list, can be removed from it this
The acquisition target of collection is completed.For example, in the present embodiment, renewal time was completed into adopting for collection more than 1 day
Collection object is deleted from acquisition target list.
Dynamic configuration component 205, is additionally operable to:Journal file set is added into the acquisition target list, by journal file
Journal file in set is as acquisition target, and sets the time interval of the journal file set poll.
This programme not only can monitor and gather the individual log file as acquisition target, can also monitor and gather day
Will file set.This programme can be monitored and gather to various forms of file sets, file set can be file,
Listed files or the form of other any set.In the present embodiment, in order to simplify description, using file set as
File illustrates.
Carry out the log data acquisition to journal file set, it is necessary first to it is configured by dynamic configuration component, it
It is monitored by dynamic monitor component, is finally acquired by log collection component afterwards.First, dynamic configuration component is triggered,
Acquisition target is configured in the dimension of file set.The journal file set for needing to be acquired is added into acquisition target
List, while using the journal file in journal file set as acquisition target, and set journal file set poll when
Between be spaced.
Afterwards, dynamic monitor component is triggered, according to the time interval of the journal file set poll, described in periodic polling
Journal file set in acquisition target list;
If occurring newly-increased journal file in the journal file set, newly-increased journal file is set as acquisition target,
And the acquisition target is added in the acquisition target list;
If the journal file in the journal file set is deleted, corresponding quilt is removed from the acquisition target list
Delete the acquisition target of journal file.
, can be according to the time of the journal file set poll set by dynamic configuration component when dynamic monitor component is triggered
Interval, the journal file set in periodic polling acquisition target list;Journal file set in poll acquisition target list
Purpose is to look at the daily record for whether occurring in newly-increased journal file or this journal file set in this journal file set
Whether file is deleted;If occurring newly-increased journal file in this journal file set, using newly-increased journal file as adopting
Collect object, and the acquisition target of corresponding newly-increased journal file is added in acquisition target list;If this journal file set
In journal file be deleted, then the acquisition target of corresponding deleted journal file is removed from acquisition target list.Pass through this
The mode of sample periodic polling file set, dynamic monitor component realize the monitoring to journal file set.
Dynamic monitor component not only monitors the journal file among file set, but also monitors any among file set
Journal file among the subclass of any level of journal file among the subclass of level, i.e. file set, is accordingly to be regarded as
Journal file among file set is monitored and the processing as acquisition target addition into acquisition target list.
For example, in the present embodiment, there are entitled Sub file set, this preferred steps in acquisition target list
This file set is monitored, the journal file quantity in this file set of periodic polling.In initial setting up, move
State configuration component by this file set add acquisition target list, and using 10 journal files in this file set as
Acquisition target.If dynamic monitor component finds there are 12 in entitled Sub file set in poll file set afterwards
Journal file, then using newly increase 2 journal files as acquisition target, and newly increase 2 acquisition targets are added into collection
List object;If found during poll file set, there was only 9 journal files in entitled Sub file set, then will be corresponding
1 acquisition target of the journal file newly deleted is deleted from acquisition target list.
The log collection component of this programme, it can trigger in for periodically whether being occurred according to acquisition target i.e. journal file
Hold change, to determine how to gather specific daily record data.It can also change to any file content does not occur, i.e., not enter
The journal file of any up-to-date of row carries out the collection of all or part of daily record data, can also be to there occurs file
Content change, that is, the journal file for having carried out up-to-date carry out the collection of all or part of daily record data.
When specifically carrying out the collection to the daily record data of acquisition target, can trigger and journal file or and file set
In the one-to-one log collection component of journal file in log collection sub-component;Corresponding one of each acquisition target is individually
Log collection sub-component, it is ensured that it is non-interference between each log collection sub-component, there is provided Log Collect System it is steady
It is qualitative;The daily record data that log collection sub-component can be newly increased with the acquisition target in taken at regular intervals list object, i.e., only enter
Daily record data incremental crawler of the hand-manipulating of needle to the journal file as acquisition target, it is not necessary to each collection of repeated acquisition every time
Object;Now, log collection sub-component needs to judge that what is the daily record data newly increased of acquisition target.
Due to the daily record data newly increased in this programme while including double implication, one kind is that do not have before in journal file
The daily record data gathered is the daily record data newly increased;Another kind be in journal file before without upload daily record data be
The daily record data newly increased.Because the daily record data of collection is all finally to gather the standard completed all to upload to be final, this
Boundary standard is used as using upload end in embodiment, that is, obtains the last actual progress daily record data upload in journal file
Position, to judge the upload performance of journal file, the daily record data newly increased not yet uploaded is also considered as what is newly increased
Daily record data, carry out the collection of actual log data.
If the data renewal end position of journal file corresponding to the acquisition target exceedes last upload end position,
Then to carrying out taken at regular intervals more than the last daily record data newly increased for uploading end position.
When the present embodiment specifically carries out the collection to the daily record data of acquisition target, it is first determined as acquisition target
It is last in the data renewal end position and journal file of journal file to upload the position terminated;If data renewal terminates
Position exceedes last upload end position, then shows the daily record data not yet uploaded be present, to terminating more than last time upload
The data not yet uploaded of position are acquired.For example, in the present embodiment, the size of the current file of an acquisition target
1000KB, end of its data renewal end position for file is first confirmd that i.e. at 1000KB, and last time i.e. nearest one
The secondary end position for completing to upload daily record data is 800KB, finds that the data renewal end position of journal file exceedes more afterwards
Last time uploads end position, then the daily record data for showing to have 1000KB-800KB=200KB not yet uploads, can be from the last time
Upload end position to rise, the 200KB daily record datas newly increased are acquired.
Fig. 6 is refer to, it is a kind of embodiment flow chart for log collection method that this programme 3rd embodiment provides.By
Log collection unit embodiment is substantially similar in log collection method embodiment, so describe fairly simple, related part
Referring to the explanation of log collection unit embodiment part.Log collection method embodiment described below is only schematic
's.The log collection method, main body is used corresponding to one, including:
Step S301:According to the demand using main body, for the acquisition target of setting, the acquisition target is gathered
Daily record data;
Step S303:Upload gathered daily record data.
Optionally, it is described according to the demand using main body in execution, for the acquisition target of setting, adopted described in collection
Before the step of collecting the daily record data of object, following step is performed:
Step S305:Using journal file as acquisition target, and the acquisition target is added into acquisition target list, and
The time interval of daily record monitoring and the time interval of log collection are set;
Step S307:The time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from described
The acquisition target for having completed collection is removed in acquisition target list.
One preferred steps of the present embodiment may be referred to Fig. 7.
Optionally, it is described to upload gathered daily record data;Including:
Limit the size that maximum single uploads data.
Optionally, it is described to upload gathered daily record data;Including:
Upload after completing, updated in end position information list is uploaded belonging to uploaded daily record data each time
Acquisition target upload end position;The wherein described end position that uploads represents that the last daily record data for completing to upload exists
End of data position in corresponding journal file.
Optionally, after completion is uploaded each time described in performing, for upper in end position information list is uploaded
After the step of acquisition target belonging to the daily record data of biography carries out uploading the renewal of end position information, following step is performed:
The upload end position information list of renewal is stored on persistent storage medium.
Optionally, it is described according to the demand using main body, for the acquisition target of setting, gather the acquisition target
Daily record data;Including:
According to the demand using main body, for the acquisition target in the acquisition target list, according to the daily record
Whether the time interval of collection and the acquisition target change, taken at regular intervals daily record data.
Optionally, the acquisition target for having completed collection includes:
Journal file, the recent renewal time of the journal file exceed given threshold, and the day away from preset time point
The data renewal end position of will file is no more than the upload end position of the journal file;
The data renewal end position represents the daily record data of the last data renewal of the journal file in institute
State the end of data position in journal file.
Optionally, the setting journal file adds acquisition target list as acquisition target, and by the acquisition target,
And the time interval of daily record monitoring and the time interval of log collection are set;Also include:
Journal file set is added into the acquisition target list, using the journal file in the journal file set as
Acquisition target, and the time interval of the journal file set poll is set.
Optionally, the time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from institute
State the acquisition target for being removed in acquisition target list and having completed collection;Also include:
According to the time interval of the journal file set poll, the daily record text in acquisition target list described in periodic polling
Part set;
If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and
The acquisition target is added in the acquisition target list;
If the journal file in the journal file set is deleted, removed from the acquisition target list with being deleted
Except the corresponding acquisition target of journal file.
Optionally, it is described whether to be changed according to the time interval and the acquisition target of the log collection, periodically
Gather daily record data;Including:
According to the time interval of the log collection, acquisition target newly increases in acquisition target list described in taken at regular intervals
Daily record data.
Optionally, acquisition target newly increases daily record data in acquisition target list described in the taken at regular intervals, including:
If the data renewal end position of the journal file exceedes the upload end position of the journal file, to institute
State the daily record data that newly increases uploaded between end position and data renewal end position and carry out taken at regular intervals.
Fig. 8 is refer to, it is a kind of embodiment schematic diagram for computing device that this programme fourth embodiment provides.Due to meter
Calculate apparatus embodiments and be substantially similar to log collection method embodiment, so describing fairly simple, related part is referring to daily record
Collecting unit and the explanation of log collection method embodiment part.Computing device embodiment described below is only schematic
's.
This programme provide a kind of computing device, be preconfigured to be it is associated with using main body, including:Processor 401
With memory 403;
The processor 401, for according to the demand using main body, for the acquisition target of setting, described in collection
The daily record data of acquisition target;Upload gathered daily record data.
Although demand this programme is disclosed as above with preferred embodiment, it is not for limiting this programme, any ability
Field technique personnel can make possible variation and modification, therefore this programme in the spirit and scope for not departing from this programme
Protection domain should be defined by the scope that this programme claim is defined.
In a typical configuration, computing device includes one or more processors (CPU), input/output interface, net
Network interface and internal memory.
Internal memory may include computer-readable medium in volatile memory, random access memory (RAM) and/or
The forms such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash RAM).Internal memory is computer-readable medium
Example.
1st, computer-readable medium can be by any side including permanent and non-permanent, removable and non-removable media
Method or technology realize that information stores.Information can be computer-readable instruction, data structure, the module of program or other numbers
According to.The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM
(SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read-only storage
(ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc are read-only
Memory (CD-ROM), digital versatile disc (DVD) or other optical storages, magnetic cassette tape, tape magnetic rigid disk storage or
Other magnetic storage apparatus or any other non-transmission medium, the information that can be accessed by a computing device available for storage.According to
Herein defines, and computer-readable medium does not include non-temporary computer readable media (transitory media), such as modulates
Data-signal and carrier wave.
2nd, it will be understood by those skilled in the art that the embodiment of this programme can be provided as method, system or computer program production
Product.Therefore, this programme can use the embodiment in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Form.Moreover, this programme can use the computer for wherein including computer usable program code in one or more can use
The computer program product that storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.)
Form.
Claims (32)
- A kind of 1. Log Collect System, it is characterised in that including:Acquisition layer equipment, including more than one log collection unit;WhereinEach described log collection unit be preconfigured to be it is associated with using main body, and according to the need using main body Ask and gather daily record data, and the daily record data of collection is uploaded to accumulation layer equipment;The accumulation layer equipment, including more than one data storage cell;WhereinEach described data storage cell is corresponding with the log collection unit, and is adopted for storing corresponding daily record Collect the daily record data that unit uploads, and the daily record data of storage is supplied to the use master associated by the log collection unit Body.
- 2. Log Collect System according to claim 1, it is characterised in that the Log Collect System, be deployed in cloud system On system.
- 3. Log Collect System according to claim 1, it is characterised in that the daily record data, including:The industry of cloud system The daily record data for service process of being engaged in.
- 4. Log Collect System according to claim 1, it is characterised in that the acquisition layer equipment, in addition to:One with On monitoring unit;WhereinEach described monitoring unit is corresponding with the log collection unit, and by the log collection unit triggers, with And it is used for:Monitor whether the log collection unit has been triggered;If it is not, then trigger the log collection unit.
- 5. Log Collect System according to claim 4, it is characterised in that the log collection unit is additionally operable to:Monitor whether the monitoring unit has been triggered;If it is not, then trigger the monitoring unit.
- 6. Log Collect System according to claim 1, it is characterised in that the data storage cell, be additionally operable to:By using cloud storage service, the daily record data of upload is stored under the account item using main body.
- 7. Log Collect System according to claim 2, it is characterised in that the Log Collect System, be deployed in described On the separate unit service server of cloud system.
- 8. Log Collect System according to claim 2, it is characterised in that more than two Log Collect Systems, It is deployed in simultaneously on same service server of the cloud system.
- A kind of 9. log collection unit, it is characterised in that be preconfigured to be it is associated with using main body, including:Log collection Component, and daily record transmitting assembly;The log collection component, for according to the demand using main body, for the acquisition target of setting, being adopted described in collection Collect the daily record data of object;The daily record transmitting assembly, for uploading gathered daily record data.
- 10. log collection unit according to claim 9, it is characterised in that also include:Dynamic configuration component, and dynamic Monitor component;The dynamic configuration component, acquisition target is added for using journal file as acquisition target, and by the acquisition target List, and the time interval of daily record monitoring and the time interval of log collection are set;The dynamic monitor component, for the time interval monitored according to the daily record, acquisition target list described in regular monitoring, And the acquisition target for having completed collection is removed from the acquisition target list.
- 11. log collection unit according to claim 9, it is characterised in that the daily record transmitting assembly, be additionally operable to:Limit the size that maximum single uploads data.
- 12. log collection unit according to claim 10, it is characterised in that the daily record transmitting assembly, be additionally operable to:Upload each time after completing, adopting belonging to uploaded daily record data is updated in end position information list is uploaded Collect the upload end position of object;The wherein described end position that uploads represents number of the last daily record data for completing to upload in corresponding journal file According to end position.
- 13. log collection unit according to claim 12, it is characterised in that the daily record transmitting assembly, be additionally operable to:By the upload end position information list of renewal, persistent storage medium is stored in.
- 14. log collection unit according to claim 10, it is characterised in that the log collection component, including:At least one log collection sub-component, whereinEach described log collection sub-component is corresponding with the acquisition target in the acquisition target list, for according to institute The demand using main body is stated, whether changes according to the time interval and the acquisition target of the log collection, periodically adopts Collect daily record data.
- 15. log collection component according to claim 14, it is characterised in that be additionally operable to:According to the number change of acquisition target in the acquisition target list, dynamically increase and decrease the log collection sub-component.
- 16. log collection unit according to claim 12, it is characterised in that the acquisition target for having completed collection Including:Journal file, the recent renewal time of the journal file exceed given threshold, and daily record text away from preset time point The data renewal end position of part is no more than the upload end position of the journal file;The data renewal end position represents the daily record data of the last data renewal of the journal file in the day End of data position in will file.
- 17. log collection unit according to claim 10, it is characterised in that the dynamic configuration component, be additionally operable to:Journal file set is added into the acquisition target list, using the journal file in the journal file set as collection Object, and the time interval of the journal file set poll is set.
- 18. log collection unit according to claim 17, it is characterised in that the dynamic monitor component, be additionally operable to:According to the time interval of the journal file set poll, the journal file collection in acquisition target list described in periodic polling Close;If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and by institute Acquisition target is stated to add in the acquisition target list;If the journal file in the journal file set is deleted, removed and deleted day from the acquisition target list The corresponding acquisition target of will file.
- 19. log collection unit according to claim 17, it is characterised in that the log collection sub-component, be used for:It is fixed Acquisition target newly increases daily record data in the phase collection acquisition target list.
- 20. log collection unit according to claim 19, it is characterised in that the log collection sub-component, be used for:If the data renewal end position of the journal file exceedes the upload end position of the journal file, on described The daily record data that newly increases passed between end position and data renewal end position carries out taken at regular intervals.
- A kind of 21. log collection method, it is characterised in that main body is used corresponding to one, including:According to the demand using main body, for the acquisition target of setting, the daily record data of the acquisition target is gathered;Upload gathered daily record data.
- 22. log collection method according to claim 21, it is characterised in that described use main body according to described performing Demand, for the acquisition target of setting, before the step of gathering the daily record data of the acquisition target, perform following step:Using journal file as acquisition target, and the acquisition target is added into acquisition target list, and set daily record to monitor Time interval and log collection time interval;The time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from the acquisition target list The acquisition target of collection has been completed in middle removal.
- 23. log collection method according to claim 21, it is characterised in that described to upload gathered daily record data; Including:Limit the size that maximum single uploads data.
- 24. log collection method according to claim 22, it is characterised in that described to upload gathered daily record data; Including:Upload each time after completing, adopting belonging to uploaded daily record data is updated in end position information list is uploaded Collect the upload end position of object;The wherein described end position that uploads represents number of the last daily record data for completing to upload in corresponding journal file According to end position.
- 25. log collection method according to claim 24, it is characterised in that uploading completion described in execution each time Afterwards, upload end position is carried out for the acquisition target belonging to the daily record data of upload in end position information list is uploaded After the step of renewal of information, following step is performed:The upload end position information list of renewal is stored on persistent storage medium.
- 26. log collection unit according to claim 22, it is characterised in that described according to the need using main body Ask, for the acquisition target of setting, gather the daily record data of the acquisition target;Including:According to the demand using main body, for the acquisition target in the acquisition target list, according to the log collection Time interval and the acquisition target whether change, taken at regular intervals daily record data.
- 27. log collection method according to claim 24, it is characterised in that the collection for having completed collection Object includes:Journal file, the recent renewal time of the journal file exceed given threshold, and daily record text away from preset time point The data renewal end position of part is no more than the upload end position of the journal file;The data renewal end position represents the daily record data of the last data renewal of the journal file in the day End of data position in will file.
- 28. log collection method according to claim 22, it is characterised in that the setting journal file is as collection pair As, and by the acquisition target add acquisition target list, and set daily record monitor time interval and log collection when Between be spaced;Also include:Journal file set is added into the acquisition target list, using the journal file in the journal file set as collection Object, and the time interval of the journal file set poll is set.
- 29. log collection method according to claim 28, it is characterised in that the time monitored according to the daily record Interval, acquisition target list described in regular monitoring, and removed from the acquisition target list and completed to adopt described in collection Collect object;Also include:According to the time interval of the journal file set poll, the journal file collection in acquisition target list described in periodic polling Close;If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and by institute Acquisition target is stated to add in the acquisition target list;If the journal file in the journal file set is deleted, removed and deleted day from the acquisition target list The corresponding acquisition target of will file.
- 30. log collection method according to claim 28, it is characterised in that the time according to the log collection Whether interval and the acquisition target change, taken at regular intervals daily record data;Including:According to the time interval of the log collection, acquisition target newly increases daily record in acquisition target list described in taken at regular intervals Data.
- 31. log collection method according to claim 30, it is characterised in that acquisition target described in the taken at regular intervals arranges Acquisition target newly increases daily record data in table, including:If the data renewal end position of the journal file exceedes the upload end position of the journal file, on described The daily record data that newly increases passed between end position and data renewal end position carries out taken at regular intervals.
- 32. a kind of computing device, be preconfigured to be it is associated with using main body, including:Processor and memory;The processor, for according to the demand using main body, for the acquisition target of setting, gathering the acquisition target Daily record data;Upload gathered daily record data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610417661.3A CN107508718B (en) | 2016-06-14 | 2016-06-14 | Log acquisition system, log acquisition unit and log acquisition method |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201610417661.3A CN107508718B (en) | 2016-06-14 | 2016-06-14 | Log acquisition system, log acquisition unit and log acquisition method |
Publications (2)
Publication Number | Publication Date |
---|---|
CN107508718A true CN107508718A (en) | 2017-12-22 |
CN107508718B CN107508718B (en) | 2021-03-02 |
Family
ID=60679104
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201610417661.3A Active CN107508718B (en) | 2016-06-14 | 2016-06-14 | Log acquisition system, log acquisition unit and log acquisition method |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN107508718B (en) |
Cited By (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108170584A (en) * | 2017-12-26 | 2018-06-15 | 广东欧珀移动通信有限公司 | log processing method, device, storage medium and terminal device |
CN109344139A (en) * | 2018-11-01 | 2019-02-15 | 浪潮电子信息产业股份有限公司 | Aggregation method and related device for operation logs of storage system |
CN110661631A (en) * | 2018-06-28 | 2020-01-07 | 中兴通讯股份有限公司 | Method, device and computer readable storage medium for collecting network element logs |
CN111143161A (en) * | 2019-12-09 | 2020-05-12 | 东软集团股份有限公司 | Log file processing method and device, storage medium and electronic equipment |
CN111723064A (en) * | 2019-03-22 | 2020-09-29 | 顺丰科技有限公司 | Log collection method and device, server and storage medium |
CN112995307A (en) * | 2021-02-09 | 2021-06-18 | 中国工商银行股份有限公司 | Log processing method, system, device, electronic equipment and storage medium |
CN113127306A (en) * | 2021-04-23 | 2021-07-16 | 苏州若拙科技有限公司 | Trusted log storage and verification system based on block chain |
CN113687872A (en) * | 2021-07-19 | 2021-11-23 | 北京鸿腾智能科技有限公司 | Selection component control method, device, storage medium and apparatus |
CN115576792A (en) * | 2022-11-24 | 2023-01-06 | 北京宝兰德软件股份有限公司 | Log collection system and method |
Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050198281A1 (en) * | 2004-02-04 | 2005-09-08 | Hon Hai Precision Industry Co., Ltd. | System and method for logging events of network devices |
CN101465765A (en) * | 2008-12-31 | 2009-06-24 | 东信和平智能卡股份有限公司 | Log system and use method thereof |
US7599939B2 (en) * | 2003-11-26 | 2009-10-06 | Loglogic, Inc. | System and method for storing raw log data |
CN103944973A (en) * | 2014-04-02 | 2014-07-23 | 北京中交兴路车联网科技有限公司 | Method and device for collecting logs |
CN102790686B (en) * | 2011-05-17 | 2015-09-16 | 浙江核新同花顺网络信息股份有限公司 | Log data acquisition method, system and server |
CN104935469A (en) * | 2015-06-30 | 2015-09-23 | 浪潮(北京)电子信息产业有限公司 | Distributive storage method and system for log information |
-
2016
- 2016-06-14 CN CN201610417661.3A patent/CN107508718B/en active Active
Patent Citations (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7599939B2 (en) * | 2003-11-26 | 2009-10-06 | Loglogic, Inc. | System and method for storing raw log data |
US20050198281A1 (en) * | 2004-02-04 | 2005-09-08 | Hon Hai Precision Industry Co., Ltd. | System and method for logging events of network devices |
CN101465765A (en) * | 2008-12-31 | 2009-06-24 | 东信和平智能卡股份有限公司 | Log system and use method thereof |
CN102790686B (en) * | 2011-05-17 | 2015-09-16 | 浙江核新同花顺网络信息股份有限公司 | Log data acquisition method, system and server |
CN103944973A (en) * | 2014-04-02 | 2014-07-23 | 北京中交兴路车联网科技有限公司 | Method and device for collecting logs |
CN104935469A (en) * | 2015-06-30 | 2015-09-23 | 浪潮(北京)电子信息产业有限公司 | Distributive storage method and system for log information |
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN108170584A (en) * | 2017-12-26 | 2018-06-15 | 广东欧珀移动通信有限公司 | log processing method, device, storage medium and terminal device |
CN110661631A (en) * | 2018-06-28 | 2020-01-07 | 中兴通讯股份有限公司 | Method, device and computer readable storage medium for collecting network element logs |
CN109344139A (en) * | 2018-11-01 | 2019-02-15 | 浪潮电子信息产业股份有限公司 | Aggregation method and related device for operation logs of storage system |
CN111723064A (en) * | 2019-03-22 | 2020-09-29 | 顺丰科技有限公司 | Log collection method and device, server and storage medium |
CN111143161A (en) * | 2019-12-09 | 2020-05-12 | 东软集团股份有限公司 | Log file processing method and device, storage medium and electronic equipment |
CN111143161B (en) * | 2019-12-09 | 2024-04-09 | 东软集团股份有限公司 | Log file processing method and device, storage medium and electronic equipment |
CN112995307A (en) * | 2021-02-09 | 2021-06-18 | 中国工商银行股份有限公司 | Log processing method, system, device, electronic equipment and storage medium |
CN112995307B (en) * | 2021-02-09 | 2023-12-12 | 中国工商银行股份有限公司 | Log processing method, system, device, electronic equipment and storage medium |
CN113127306A (en) * | 2021-04-23 | 2021-07-16 | 苏州若拙科技有限公司 | Trusted log storage and verification system based on block chain |
CN113687872A (en) * | 2021-07-19 | 2021-11-23 | 北京鸿腾智能科技有限公司 | Selection component control method, device, storage medium and apparatus |
CN113687872B (en) * | 2021-07-19 | 2024-03-29 | 三六零数字安全科技集团有限公司 | Selection component control method, device, storage medium and apparatus |
CN115576792A (en) * | 2022-11-24 | 2023-01-06 | 北京宝兰德软件股份有限公司 | Log collection system and method |
Also Published As
Publication number | Publication date |
---|---|
CN107508718B (en) | 2021-03-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107508718A (en) | Log Collect System, log collection unit and log collection method | |
US7418489B2 (en) | Method and apparatus for applying policies | |
US11593302B2 (en) | Methods, systems, and computer readable mediums for implementing a data protection policy for a transferred enterprise application | |
CN106202346B (en) | A kind of data load cleaning engine, scheduling and storage system | |
DE112011103666B4 (en) | Storage management in cluster data processing systems | |
CN109067828A (en) | Based on the more cluster construction methods of Kubernetes and OpenStack container cloud platform, medium, equipment | |
CN107567696A (en) | The automatic extension of resource instances group in computing cluster | |
CN102523101B (en) | Machine manager service fabric | |
CN110764871B (en) | Cloud platform-based mimicry application packaging and control system and method | |
WO2017059294A1 (en) | Systems and methods for detecting vulnerabilities and privileged access using cluster movement | |
CN109634716A (en) | The OpenStack virtual machine High Availabitity management end device and management method of anti-fissure | |
CN107688611B (en) | Saltstack-based Redis key value management system and method | |
CN102857371B (en) | A kind of dynamic allocation management method towards group system | |
CN107547273A (en) | A kind of support method and system of power system virtual instance High Availabitity | |
CN109614201A (en) | The OpenStack virtual machine high-availability system of anti-fissure | |
CN108171050A (en) | The fine granularity sandbox strategy method for digging of linux container | |
CN107908651A (en) | A kind of auditing method of distributed type assemblies | |
CN205507061U (en) | Radar is synthesized and is taken notes appearance | |
DE102016203598A1 (en) | COMMUNITY COLLECTION OF DIAGNOSTIC DATA OF SOFTWARE PROGRAMS | |
CN105096014A (en) | Method and system for recording work operation condition remotely | |
CN109981350A (en) | A kind of continual upgrade method of cloud service and system | |
CN105759247A (en) | Radar comprehensive recorder and radar data recording method | |
CN108228390A (en) | Data return shelves method and device | |
CN108021431B (en) | Web data interaction based Hive management method and system | |
CN109359205A (en) | A kind of remote sensing image cutting method and equipment based on geographical grid |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 1248413 Country of ref document: HK |
|
GR01 | Patent grant | ||
GR01 | Patent grant |