CN107508718A - Log Collect System, log collection unit and log collection method - Google Patents

Log Collect System, log collection unit and log collection method Download PDF

Info

Publication number
CN107508718A
CN107508718A CN201610417661.3A CN201610417661A CN107508718A CN 107508718 A CN107508718 A CN 107508718A CN 201610417661 A CN201610417661 A CN 201610417661A CN 107508718 A CN107508718 A CN 107508718A
Authority
CN
China
Prior art keywords
acquisition target
daily record
log collection
journal file
log
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201610417661.3A
Other languages
Chinese (zh)
Other versions
CN107508718B (en
Inventor
宋军
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Alibaba Group Holding Ltd
Original Assignee
Alibaba Group Holding Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Ltd filed Critical Alibaba Group Holding Ltd
Priority to CN201610417661.3A priority Critical patent/CN107508718B/en
Publication of CN107508718A publication Critical patent/CN107508718A/en
Application granted granted Critical
Publication of CN107508718B publication Critical patent/CN107508718B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/06—Management of faults, events, alarms or notifications
    • H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Environmental & Geological Engineering (AREA)
  • Debugging And Monitoring (AREA)

Abstract

This programme provides a kind of Log Collect System, including：Acquisition layer equipment, including more than one log collection unit；Log collection unit is preconfigured to be associated with using main body described in each of which, and gathers daily record data using the demand of main body according to described, and the daily record data of collection is uploaded into accumulation layer equipment；The accumulation layer equipment, including more than one data storage cell；Data storage cell described in each of which is corresponding with the log collection unit, and the daily record data uploaded for storing corresponding log collection unit, and the daily record data of storage is supplied into the use main body associated by the log collection unit.This programme provides a kind of log collection unit and a kind of log collection method simultaneously.This programme can solve the problems, such as that the distributed information log acquisition system of prior art lacks collection flexibility, system cost is too high and system deployment is excessively complicated.

Description

Log Collect System, log collection unit and log collection method
Technical field
This programme is related to daily record data processing technology field, and in particular to Log Collect System.This programme further relates to daily record Collecting unit and log collection method.
Background technology
With the rapid development of cloud computing, largely occur one after another for the exclusive cloud system of specialized business scenario, this its In outstanding person include PaaS (platform i.e. service) and SaaS (software is service) etc..These cloud systems are typically all using main body Corresponding cloud service is provided on the premise of mandate by privately owned cluster；But due to using for main body, these cloud systems Privately owned cluster be a black box, i.e. the bottom-up information (such as virtual machine and storage) of cloud service is using main body is invisible and nothing Method control.In this case, the cluster running status of cloud system and the business procedure of related bottom-up information can be reflected Daily record is vital；Cloud system is needed to provide the accurately daily record data with real-time business procedure using main body, to assist Whether it checks cloud service safe and reliable.
At present, above-mentioned daily record number can be met by proposing several distributed massive logs acquisition system in the prior art According to the demand of collection, still, for system architecture, the distributed Log Collect System of these prior arts is all quite multiple It is miscellaneous, it is generally divided into three layers：
Include the log collection layer of multiple log collection units；
Include the data gathering layer of multiple data collection servers；
Include the data storage layer of multiple data storage servers.
Each log collection unit wherein in log collection layer is all according to system when gathering specific daily record data Preset directly to carry out, it is impossible to make selection and the tune of collection according to the final real needs using main body of daily record data It is whole；The acquisition mode of this fixation often results in the waste of collection lack of targeted and gathered data, while also requirement must be set Put data gathering layer, supporting corresponding data collection server, come sort daily record data that log collection unit gathered and Determine gathered daily record data relative to the ownership using main body.And completed in the sorting and determination operation of data gathering layer Afterwards, then with data gathering layer by the data storage of collection to data storage layer.
From cost, the distributed acquisition system of prior art due to being divided into three structure sheafs in the vertical, it is necessary to Data collection server is disposed to sort daily record data and determine the ownership of daily record data gathered；In the horizontal can not be in separate unit Disposed on server, it is necessary to safeguard data collection server cluster and data storage server cluster.Such prior art The lower deployment cost of distributed information log acquisition system is at a relatively high.
In actual application, the daily record data amount of many cloud systems using subject demandses is simultaneously little, also will not be right Daily record data carries out the analysis and excavation of depth, normally only the specific complementary and daily record data of Process Character of demand.For example, make When handling cloud service with big data, user performs some operations specific, it is necessary to check that operations specific performs in time in cloud system Output journal, to ensure that when operations specific malfunctions error reason can be viewed.Use main body to these daily record datas For, the distributed information log acquisition system of prior art lacks collection flexibility, and the cost of a full set of system is too high, and Account Dept Administration is also excessively complicated.
The content of the invention
This programme provides a kind of Log Collect System, a kind of log collection unit and a kind of log collection method, with solution Certainly the distributed information log acquisition system of prior art lacks collection flexibility, system cost is too high and system deployment is excessively complicated The problem of.
This programme provides a kind of Log Collect System, including：
Acquisition layer equipment, including more than one log collection unit；Wherein
Each described log collection unit is preconfigured to be associated with using main body, and uses main body according to described Demand and gather daily record data, and the daily record data of collection is uploaded to accumulation layer equipment；
The accumulation layer equipment, including more than one data storage cell；Wherein
Each described data storage cell is corresponding with the log collection unit, and for storing corresponding day The daily record data that will collecting unit uploads, and the daily record data of storage is supplied to associated by the log collection unit made Use main body.
Optionally, the Log Collect System, is deployed on cloud system.
Optionally, the daily record data, including：The daily record data of the business service process of cloud system.
Optionally, the acquisition layer equipment, in addition to：More than one monitoring unit；Wherein
Each described monitoring unit is corresponding with the log collection unit, and is touched by the log collection unit Hair, and be used for：Monitor whether the log collection unit has been triggered；
If it is not, then trigger the log collection unit.
Optionally, the log collection unit is additionally operable to：
Monitor whether the monitoring unit has been triggered；
If it is not, then trigger the monitoring unit.
Optionally, the data storage cell, is additionally operable to：
By using cloud storage service, the daily record data of upload is stored under the account item using main body.
Optionally, the Log Collect System, it is deployed on the separate unit service server of the cloud system.
Optionally, more than two Log Collect Systems, while it is deployed in the same business clothes of the cloud system It is engaged on device.
This programme also provides a kind of log collection unit, be preconfigured to be it is associated with using main body, including：Daily record is adopted Collect component, and daily record transmitting assembly；
The log collection component, for according to the demand using main body, for the acquisition target of setting, gathering institute State the daily record data of acquisition target；
The daily record transmitting assembly, for uploading gathered daily record data.
Optionally, in addition to：Dynamic configuration component, and dynamic monitor component；
The dynamic configuration component, for using journal file as acquisition target, and the acquisition target is added and gathered List object, and the time interval of daily record monitoring and the time interval of log collection are set；
The dynamic monitor component, for the time interval monitored according to the daily record, acquisition target described in regular monitoring List, and the acquisition target for having completed collection is removed from the acquisition target list.
Optionally, the daily record transmitting assembly, is additionally operable to：
Limit the size that maximum single uploads data.
Optionally, the daily record transmitting assembly, is additionally operable to：
Upload after completing, updated in end position information list is uploaded belonging to uploaded daily record data each time Acquisition target upload end position；
The wherein described end position that uploads represents the last daily record data for completing to upload in corresponding journal file End of data position.
Optionally, the daily record transmitting assembly, is additionally operable to：
By the upload end position information list of renewal, persistent storage medium is stored in.
Optionally, the log collection component, including：
At least one log collection sub-component, wherein
Each described log collection sub-component is corresponding with the acquisition target in the acquisition target list, for root According to the demand using main body, whether changed according to the time interval and the acquisition target of the log collection, it is fixed Phase gathers daily record data.
Optionally, it is additionally operable to：
According to the number change of acquisition target in the acquisition target list, dynamically increase and decrease the log collection sub-component.
Optionally, the acquisition target for having completed collection includes：
Journal file, the recent renewal time of the journal file exceed given threshold, and the day away from preset time point The data renewal end position of will file is no more than the upload end position of the journal file；
The data renewal end position represents the daily record data of the last data renewal of the journal file in institute State the end of data position in journal file.
Optionally, the dynamic configuration component, is additionally operable to：
Journal file set is added into the acquisition target list, using the journal file in the journal file set as Acquisition target, and the time interval of the journal file set poll is set.
Optionally, the dynamic monitor component, is additionally operable to：
According to the time interval of the journal file set poll, the daily record text in acquisition target list described in periodic polling Part set；
If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and The acquisition target is added in the acquisition target list；
If the journal file in the journal file set is deleted, removed from the acquisition target list with being deleted Except the corresponding acquisition target of journal file.
Optionally, the log collection sub-component, is used for：Acquisition target is new in acquisition target list described in taken at regular intervals Increase daily record data.
Optionally, the log collection sub-component, is used for：
If the data renewal end position of the journal file exceedes the upload end position of the journal file, to institute State the daily record data that newly increases uploaded between end position and data renewal end position and carry out taken at regular intervals.
This programme also provides a kind of log collection method, and main body is used corresponding to one, including：
According to the demand using main body, for the acquisition target of setting, the daily record data of the acquisition target is gathered；
Upload gathered daily record data.
Optionally, it is described according to the demand using main body in execution, for the acquisition target of setting, adopted described in collection Before the step of collecting the daily record data of object, following step is performed：
Using journal file as acquisition target, and the acquisition target is added into acquisition target list, and daily record is set The time interval of monitoring and the time interval of log collection；
The time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from the acquisition target The acquisition target for having completed collection is removed in list.
Optionally, it is described to upload gathered daily record data；Including：
Limit the size that maximum single uploads data.
Optionally, it is described to upload gathered daily record data；Including：
Upload after completing, updated in end position information list is uploaded belonging to uploaded daily record data each time Acquisition target upload end position；
The wherein described end position that uploads represents the last daily record data for completing to upload in corresponding journal file End of data position.
Optionally, after completion is uploaded each time described in performing, for upper in end position information list is uploaded After the step of acquisition target belonging to the daily record data of biography carries out uploading the renewal of end position information, following step is performed：
The upload end position information list of renewal is stored on persistent storage medium.
Optionally, it is described according to the demand using main body, for the acquisition target of setting, gather the acquisition target Daily record data；Including：
According to the demand using main body, for the acquisition target in the acquisition target list, according to the daily record Whether the time interval of collection and the acquisition target change, taken at regular intervals daily record data.
Optionally, the acquisition target for having completed collection includes：
Journal file, the recent renewal time of the journal file exceed given threshold, and the day away from preset time point The data renewal end position of will file is no more than the upload end position of the journal file；
The data renewal end position represents the daily record data of the last data renewal of the journal file in institute State the end of data position in journal file.
Optionally, the setting journal file adds acquisition target list as acquisition target, and by the acquisition target, And the time interval of daily record monitoring and the time interval of log collection are set；Also include：
Journal file set is added into the acquisition target list, using the journal file in the journal file set as Acquisition target, and the time interval of the journal file set poll is set.
Optionally, the time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from institute State the acquisition target for being removed in acquisition target list and having completed collection；Also include：
According to the time interval of the journal file set poll, the daily record text in acquisition target list described in periodic polling Part set；
If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and The acquisition target is added in the acquisition target list；
If the journal file in the journal file set is deleted, removed from the acquisition target list with being deleted Except the corresponding acquisition target of journal file.
Optionally, it is described whether to be changed according to the time interval and the acquisition target of the log collection, periodically Gather daily record data；Including：
According to the time interval of the log collection, acquisition target newly increases in acquisition target list described in taken at regular intervals Daily record data.
Optionally, acquisition target newly increases daily record data in acquisition target list described in the taken at regular intervals, including：
If the data renewal end position of the journal file exceedes the upload end position of the journal file, to institute State the daily record data that newly increases uploaded between end position and data renewal end position and carry out taken at regular intervals.
This programme also provides a kind of computing device, be preconfigured to be it is associated with using main body, including：Processor and deposit Reservoir；
The processor, for according to the demand using main body, for the acquisition target of setting, gathering the collection The daily record data of object；Upload gathered daily record data.
Compared with prior art, this programme has advantages below：
This programme, can be corresponding directly against being carried out using main body by the way that log collection unit is associated with using main body Daily record data collection, it is not necessary to the participation of data collection server, in vertical structure only have more simple daily record adopt Two layers of collection and data storage；Meanwhile this programme can use the supporting data storage cell of a log collection unit Form a complete Log Collect System, it is not necessary to which too fat to move distributed type assemblies, Log Collect System can be disposed easily Used in enterprising exercise of separate unit service server.
This programme provide Log Collect System can according to using main body particular demands, flexible configuration acquisition target, Implement targetedly log data acquisition, improve the efficiency of log data acquisition, it also avoid collection redundant data and cause Waste.
The Log Collect System that this programme provides, due to simple in construction, only it can be disposed on separate unit service server, It can be disposed on any number of server, autgmentability is relatively good；Meanwhile the Log Collect System of this programme need not service The support of device cluster, the cost of system deployment is than relatively low.
The daily record data that the Log Collect System of this programme is gathered, both for specific using main body, these daily records Data will not mutually mix with other any daily record datas using main body, while these daily record datas are also stored in for specific Using the memory space of main body, the security and privacy of the data of Log Collect System be all higher.
The Log Collect System of this programme can also carry out storing daily record data using independent cloud storage service, so as to enter one Step has simplified the structure of Log Collect System, reduces the cost of system deployment, also ensure that the security of daily record data and hidden Private.
Brief description of the drawings
Fig. 1 is a kind of structural representation of the distributed information log acquisition system for prior art that this programme provides；
Fig. 2 is a kind of structural representation for Log Collect System that this programme first embodiment provides；
Fig. 3 is the signal that a kind of log collection unit that this programme first embodiment provides mutually monitors with monitoring unit Figure；
Fig. 4 is a kind of structural representation for log collection unit that this programme second embodiment provides；
Fig. 5 is a kind of schematic diagram for upload end position information list that this programme second embodiment provides；
Fig. 6 is a kind of flow chart for log collection method that this programme 3rd embodiment provides；
Fig. 7 is a kind of flow chart for log collection method preferred steps that this programme 3rd embodiment provides；
Fig. 8 is a kind of schematic diagram for computing device that this programme fourth embodiment provides.
Embodiment
Many details are elaborated in the following description in order to fully understand this programme.But this programme can be with Much it is different from other manner described here to implement, those skilled in the art can be in the situation without prejudice to this programme intension Under do similar popularization, therefore this programme is not limited by following public specific implementation.
In this programme, a kind of Log Collect System, a kind of log collection unit, a kind of log collection side are each provided Method and a kind of computing device.Below in conjunction with the accompanying drawings, it is described in detail one by one in the following embodiments.
Fig. 2 is refer to, it is a kind of structural representation for Log Collect System that this programme first embodiment provides.It is described Log Collect System, including：
Acquisition layer equipment 101, including more than one log collection unit 101-1；Wherein, each described log collection Unit 101-1 is preconfigured to be associated with using main body, and gathers daily record data using the demand of main body according to described, And the daily record data of collection is uploaded to accumulation layer equipment 103；
The accumulation layer equipment 103, including more than one data storage cell 103-1；Wherein, each described data Memory cell 103-1 is corresponding with the log collection unit 101-1, and for storing corresponding log collection unit The daily record data that 101-1 is uploaded, and the daily record data of storage is supplied to associated by the log collection unit 101-1 made Use main body.
The Log Collect System of this programme, include the acquisition layer equipment 101 of multiple log collection units and there are more numbers According to the accumulation layer equipment 103 of memory cell；Wherein, each log collection unit is by the use with particular log data in advance Main body is associated, and gathers using the demand of the specific usage log data of main body according to this daily record data and upload and adopt The daily record data of collection to accumulation layer equipment 103 data storage cell；Each data storage cell both corresponds to particular log Collecting unit, and the daily record data gathered dedicated for particular log collecting unit corresponding to storage, and the day by storage Will data are supplied to the use main body associated by this log collection unit.
In this programme is exactly the actually user of daily record data, including logical outside Log Collect System using main body User that the Log Collect System often said is serviced and other need to use the equipment or facility of daily record data.In this programme In, each log collection unit, which associates, specifically uses main body；Log collection unit is being configured to associate specific use master It is as specific to use the exclusive use of main body after body, it is impossible to be configured association again and use main body to other.For example, in this implementation In example, as shown in Fig. 2 each corresponding use respectively of log collection unit 1, log collection unit 2 and log collection unit 3 Main body, i.e. daily record collecting unit 1 are corresponding to use main body 1, and each corresponding use is led for log collection unit 2 and log collection unit 3 Body 2, and use main body 2 then to correspond to two log collection units.
Log collection unit in this programme, for the collection and upload of daily record data to be embodied.The of this programme Log collection unit will will be described in detail in two embodiments.
Daily record data in this programme, can be the daily record data of the business service process of cloud system, i.e. cloud system is carrying During business service of the supply using main body cloud system, about the business procedure such as specifically being calculated, being handled and being stored Daily record data；Daily record data or use other required any type of data of main body in this programme, such as industry Some specific service state of business server etc..Daily record data in this programme is also not excluded for applying and is deployed in addition to cloud system Situation, for example, it is also possible to be the daily record data on separate unit service server.
Log Collect System in this programme, can be the Log Collect System being deployed on cloud system.It can also be portion Administration is applied to other distributed and non-distributed network system even single server systems in addition to cloud system.In this reality Apply in example, for the purpose of simplifying the description, the Log Collect System being preferably deployed on cloud system.Generally, in this programme Also it is the use main body of cloud system simultaneously using main body, the business processing service of cloud system offer is provided.
It please also refer to Fig. 1, be in place of the difference of the distributed information log acquisition system of this programme and prior art：
First, each log collection unit of real data collection, quilt are carried out in the collecting device layer 101 of this programme Be configured to using main body it is associated (in one embodiment, a log collection unit can be associated with one and use main body, In yet another alternative embodiment, a log collection unit can be associated with it is multiple use main body, replaced at also one real Apply in example, multiple log collection units can be associated with one and use main body)；And the daily record of the Log Collect System of prior art Collecting unit does not establish corresponding relation with any one using main body.One of skill in the art will understand that at least it is based on For each log collection unit of this programme with the clear and definite incidence relation established using main body, log collection unit can With the real needs using main body to daily record data associated according to it, flexible configuration acquisition target, so as to implement targetedly Log data acquisition, improve collecting efficiency, avoid gathering unwanted data；And the day of the Log Collect System of prior art The log data acquisition of will collecting unit is that system is preset and changeless, it is impossible to according to the specific of specifically used main body Demand carrys out the log data acquisition of flexible customization log collection unit；
Second, because each log collection unit of this programme using main body with establishing clear and definite incidence relation, Then further log collection unit can establish corresponding relation with this using the data storage cell of main body, and can directly by The data gathered are uploaded to corresponding data storage cell by log collection unit, thus, the Log Collect System of this programme The prerequisite data gathering layer of institute in the prior art is eliminated in vertical structure, greatly simplify the structure of whole system, Realize light-weighted system deployment effect.By contrast, log collection unit of the prior art does not make with any one Corresponding relation is established with main body (and the data storage cell of main body is used for this), it is therefore necessary to disposes data gathering layer The daily record data and confirm which gathered attribution data use main body in that sorting log collection unit is gathered, so make Obtaining the structure of whole system tends to be complicated, improves cost of implementation；
3rd, because each log collection unit of this programme using main body and/or data storage cell with being established Association or corresponding relation, then the daily record data that log collection unit is gathered be specific to it is specific use main body, and adopted by daily record Collection unit is directly uploaded in the specific data storage cell using main body, thus, in daily record data from collecting storage In whole process, the specific daily record data using main body will not mutually be mixed with other using the daily record data of main body, such day The security and privacy of will data can be guaranteed；And in the Log Collect System of prior art, different daily records Data (may belong to different use main bodys) need to be divided via data gathering layer in collection, collection and storing process Pick and confirm, therefore be mingled in and together, be stolen which increase daily record data, replicated, the possibility of malice forwarding, be unfavorable for Data-privacy is protected.
It should be noted that because each log collection unit of this programme is with using main body and/or data storage cell Establish association or corresponding relation, the flowing of the daily record data of collection in the Log Collect System of this programme can only be longitudinal direction not It can be horizontal, i.e., data storage cell can only be uploaded to from log collection unit, then use is supplied to by data storage cell Main body.For example, the daily record data gathered from log collection unit 2, is only possible to upload to data storage cell 2, then provides Give use main body 2.This daily record data can not possibly be supplied to data storage cell 3.
As shown in figure 3, the acquisition layer equipment 101 in this programme, in addition to it can include multiple log collection units, also More than one monitoring unit 101-3 can be included；Wherein, each described monitoring unit 101-3 with the log collection Unit 101-1 is corresponding, and is triggered by the log collection unit 101-1.How many Log Collect System can is corresponding Supporting how many individual monitoring units.
Monitoring unit is mainly used in monitoring whether log collection unit 101-1 has been triggered；If log collection unit It is not triggered, then triggers log collection unit 101-1.Moreover, log collection unit 101-1 can also be monitored in turn This monitoring unit 101-3, for monitoring whether this monitoring unit 101-3 has been triggered；If monitoring unit 101-3 does not have It is triggered, then triggers monitoring unit 101-3.
Monitoring and log collection unit monitoring to monitoring unit of the monitoring unit to log collection unit, can be monitoring Monitoring in real time or at regular intervals interval carry out periodicity monitoring once；Judge log collection unit or The standard that monitoring unit is not triggered can be determined flexibly.For example, monitoring unit can at set intervals, periodically 1 monitoring to log collection unit was carried out every 1 minute；The mark such as whether can be acquired by log collection unit Standard judges whether log collection unit is triggered.
The data storage cell 103-1 in accumulation layer equipment 103 in this programme, corresponding to a specific log collection Unit, that is, specifically use main body corresponding to one.Data storage cell can use a variety of storages in the specific implementation Mode and storage medium, to store the daily record data that corresponding log collection unit is gathered, for example, directly in single server On hard disk on store, either storage network hard disc in a local network or storage is what Log Collect System was disposed Under the cloud storage account item that same cloud system provides, etc..Allow for obtaining day for being stored of data storage cell using main body Will data, either using which kind of concrete mode, such as directly read from home server hard disk or this LAN network hard disc, Or downloaded from network etc..
The data storage cell 103-1 in accumulation layer equipment 103 in this programme, is additionally operable to take by using cloud storage Business, the daily record data of upload is stored under the account item using main body.
When carrying out data storage using the cloud storage service for providing separate storage account, main body can also be used by one Daily record data is all stored in ownership, and this is used under the cloud storage account item of main body.Now, cloud storage can on the one hand be utilized Independence and privacy between system account, ensure the safety of the daily record data of storage；On the other hand daily record can also be utilized Cloud storage service outside acquisition system, further simplify the structure of Log Collect System；This is substantially equivalent to by daily record Contracted out outside the accumulation layer equipment of acquisition system, implement the work(of the accumulation layer equipment of Log Collect System using cloud storage service Can, the whole Log Collect System of lightweight.Certainly, in one embodiment, can be by a daily record data using main body It is stored in and belongs to this one or more cloud storage account using a cloud storage system of main body；It is real in another replacement Apply in example, a daily record data using main body can also be stored in multiple accounts of multiple cloud storage systems.The present embodiment It is preferred that the cloud storage service provided in itself by the cloud system of deployment by Log Collect System is provided, the efficiency of such data storage Compare high.For example, in the present embodiment, possess independent cloud in the cloud system that Log Collect System is disposed using main body 1 Authority is stored, corresponding data storage cell can is using master by using the cloud storage account and password using main body 1 The daily record data that corresponding log collection unit uploads is stored under the cloud storage account item of body 1.
Log Collect System in this programme, horizontal extension can be carried out.Can be according to the specific of log data acquisition Need, corresponding one or more multiple Log Collect Systems using main body are deployed in multiple service servers of cloud system On, it is this extending transversely to be virtually limitless；Log Collect System in this programme, horizontal simplify can also be carried out. Can be a daily record by a corresponding minimum Log Collect System using main body according to the specific needs of log data acquisition Collecting unit and a data storage cell are deployed directly on the separate unit service server of cloud system；Or corresponding two are made With more than two log collection units of main body, it is deployed on same service server of cloud system.For example, in this implementation In example, the Log Collect System in Fig. 2 is to be deployed on separate unit service server, wherein log collection unit 1 and daily record Collecting unit 2,3 is respectively using main body 1 and using main 2 collection daily record data.
Fig. 4 is refer to, it is a kind of structural representation for log collection unit that this programme second embodiment provides.We The log collection unit 101 of case, including：Log collection component 201, and daily record transmitting assembly 203；
The log collection component 201, for according to the demand using main body, for the acquisition target of setting, adopting Collect the daily record data of the acquisition target.
Log collection component 201, the use according to corresponding to log collection unit to be used among log collection unit 101 Main body, for these journal files as acquisition target, implements the collection of specific daily record data to the demand of journal file.
The daily record transmitting assembly 203, for uploading gathered daily record data.
Daily record transmitting assembly 203, it is used to upload what log collection component 201 was gathered among log collection unit 101 Daily record data.
It can be seen that a minimum log collection unit, can include a log collection component and a daily record upload group Part；If add a data storage cell, it is possible to form a minimum Log Collect System.
The log collection unit of this programme, in addition to：Dynamic configuration component 205, and dynamic monitor component 207.
The dynamic configuration component 205, for using journal file as acquisition target, and the acquisition target is added and adopted Collect list object, and the time interval of daily record monitoring and the time interval of log collection are set.
Dynamic configuration component 205 among the log collection unit of this programme, for using journal file as acquisition target, And this acquisition target is added into acquisition target list, i.e., newly-installed acquisition target is added into acquisition target list, Yi Jishe Put the time interval of daily record monitoring and the time interval of log collection is set.Progress daily record can be set to adopt by this component 205 Basic parameter needed for collection, and monitor and manage collection activity by the basic parameter of these settings.
The acquisition target of this programme can be diversified forms, can be the daily record text of the business service process of cloud system During the business service using main body cloud system is supplied to, record is relevant specifically to be calculated, is handled for part, i.e. cloud system With storage etc. business procedure journal file；Acquisition target in this programme or using main body it is required other The data for type of anticipating, such as some specific service state of service server etc..Acquisition target in this programme is also not excluded for should With the situation being deployed in addition to cloud system, for example, it is also possible to be the journal file on separate unit service server.In the present embodiment In, it is specific journal file corresponding to acquisition target to simplify description, i.e., each acquisition target is exactly to add collection pair As a journal file in list；It is being to confirm that this journal file is shape to be collected using journal file as acquisition target State, do not make any change or mark in itself to journal file, journal file is simply included in classification to be collected.Specific During implementation, file name asterisk wildcard can be set, to support to meet that the journal file of this title asterisk wildcard is used as acquisition target.
Acquisition target list contains the acquisition target that log collection unit is acquired, that is, contains state to be collected Journal file, daily record monitor component determine its specific acquisition target monitored according to the content of acquisition target list, and daily record is adopted Collection component similarly determines its specific journal file gathered according to acquisition target list.For example, 20 journal files are made Acquisition target list is added for acquisition target, that is, it is state to be collected to confirm this 20 journal files.
The time interval of daily record monitoring and the time interval of log collection, are all the automatic or manual tools set manually of system The period of the specific duration of body, typically in seconds.The two time intervals are all for follow-up daily record monitoring step and day The step of will gathers does parameter preparation；Time interval it is shorter, daily record monitoring and log collection are more real-time.For example, set The time interval of daily record monitoring is 30 seconds, and the time interval for setting log collection is 10 seconds.
The dynamic monitor component 207, for the time interval monitored according to the daily record, collection pair described in regular monitoring As list, and the acquisition target for having completed collection is removed from the acquisition target list.
Dynamic monitor component 207 is used for the time interval monitored according to the daily record set by dynamic configuration component 205, periodically The acquisition target list set by dynamic configuration component 205 is monitored, and is removed from this acquisition target list and has completed to adopt The acquisition target of collection；Dynamic monitor component 207 is additionally operable to specific monitoring collection list object, and removes and complete adopting for collection Collect object, to prevent the situation that acquisition target is more and more, swarms in acquisition target list.
Dynamic monitor component 207 is the time interval for being monitored according to the daily record set by dynamic configuration component 205, fixed Phase triggering, i.e., the time interval monitored every the daily record of one section of setting, once dynamic monitor component 207 is triggered, and so on.
Adopted in the design parameter needed for the setting log collection of dynamic configuration component 205 and the monitoring of dynamic monitor component 207 After collecting list object, the log collection component 201 of this programme can be further refined.Log collection component 201, if in addition to A dry log collection sub-component 201-1, wherein, each described log collection sub-component with the acquisition target list One acquisition target is corresponding, for according to the demand using main body, time interval and institute according to the log collection State whether acquisition target changes, taken at regular intervals daily record data.
Log collection component 201 includes one or more log collection sub-component 201-1 for corresponding to single acquisition target； Wherein, each described log collection sub-component mutually corresponds with the acquisition target in the acquisition target list, adopts Collect that how many acquisition target in list object just corresponds to the how many individual log collection sub-components of triggering；Each log collection subgroup Part is all used for using the real needs of main body corresponding to the log collection unit belonging to, according to the time of the log collection of setting Whether interval and acquisition target change in file level, the corresponding daily record data of taken at regular intervals.Such as in the present embodiment, As shown in figure 3, log collection component includes 3 log collection sub-components, they correspond to 3 daily records as acquisition target respectively File, carry out the collection of daily record data.
Log collection component 201 also includes：According to the number change of acquisition target in the acquisition target list, dynamic increases Subtract the log collection sub-component.If the quantity increase of acquisition target, log collection component dynamic in acquisition target list The quantity of log collection sub-component is adjusted, it is corresponding to increase log collection sub-component, make increased log collection sub-component and increase Acquisition target it is corresponding；If the quantity of acquisition target is reduced in acquisition target list, log collection component dynamic adjusts The quantity of log collection sub-component, reduces corresponding log collection sub-component, and corresponding remove corresponds to by from acquisition target list The log collection sub-component of the acquisition target of removal.The quantity of log collection sub-component is with collection pair in acquisition target list The number change of elephant and dynamic change.
The daily record transmitting assembly 203 of this programme, is additionally operable to：Limit the size that maximum single uploads data.
Daily record transmitting assembly can follow a upload when for uploading daily record data that log collection component is gathered The limitation of bandwidth, i.e. maximum single upload size of data without departing from the restriction bandwidth upper limit.Set uploading bandwidth limitation be in order to Prevent the network traffics of service server that Log Collect System causes to be disposed from blocking, ensure the stability of system operation.
For example, if limiting maximum single uploads the size of data as 500KB, it is assumed that corresponds to some and uses main body, daily record Collecting unit is directed to some acquisition target, the new daily record data for gathering 600KB, then when triggering daily record transmitting assembly, Zhi Nengyi Secondary upload 500KB daily record data.
Daily record transmitting assembly 203, is additionally operable to：Upload each time after completing, in end position information list is uploaded The upload end position of acquisition target belonging to the uploaded daily record data of renewal；Wherein described upload end position represents nearest End of data position of the daily record data for once completing to upload in corresponding journal file.
Daily record transmitting assembly 203, can also be each when for uploading daily record data that log collection component is gathered After secondary upload is completed, in end position information list is uploaded, enter for the acquisition target belonging to the daily record data of upload Row uploads the renewal of end position.
The upload end position of acquisition target, it is to be directed to particular acquisition object, the last time completes upload daily record data and existed As the Data Position in the journal file of particular acquisition object, this upload end position can determine to have completed to upload and Position boundary of the daily record data not yet uploaded in journal file.For example, the log file size of an acquisition target For 1000KB.In one embodiment, the last data for completing to upload daily record data are determined by uploading end position End position is at 800KB, then can determine in this journal file, upload 800KB daily record datas, is also had 1000KB-800KB=200KB daily record datas not yet upload.
End position information list is uploaded, is the summary table for the upload end position information for listing acquisition target, it specify that The last end position for uploading data in journal file of each acquisition target.Upload end position information list tool Body form is exactly one-to-one two-dimensional table, refer to Fig. 5, and it is that the upload that this programme second embodiment provides terminates The content schematic diagram of positional information list.It is the daily record text as acquisition target to upload the row in end position information list Part, another row are that the last of the corresponding journal file of record uploads the particular location terminated.
Because the other assemblies outside the daily record uploading unit of this programme need to obtain or update the upload knot of acquisition target Beam position information, after triggering daily record transmitting assembly each time and completing to upload daily record data, daily record transmitting assembly can also be used In the upload knot that corresponding acquisition target to the acquisition target belonging to daily record data, is carried out in end position information list is uploaded The renewal of beam position information.
For example, in the present embodiment, a log file size as acquisition target is 1000KB, the last time is namely The end position that the last time completes to upload daily record data is 800KB, i.e., in end position information list is uploaded, record uploads End position is 800KB.Continue to acquire 200KB daily record datas again since 800KB positions afterwards, correspondingly upload again 150KB daily record datas among 200KB, then it is 1000KB for this file size in end position information list is uploaded Acquisition target, it is 950KB to update it to upload end position.
Completed in triggering daily record transmitting assembly after renewal uploads end position information list, daily record transmitting assembly can be with For by the upload end position information list of renewal, being stored in persistent storage medium.
By carrying out persistent storage to the upload end position information list of renewal, for example, in the correlation of local hard drive Storage catalogue, it can prevent that end position information is uploaded caused by service server powers off the machine of delaying suddenly loses, so as to avoid Repeat to upload daily record data, and improve the stability and efficiency of Log Collect System.
The acquisition target for having completed collection includes：Journal file, during the recent renewal of the journal file Spacing preset time point exceedes given threshold, and the data renewal end position of the journal file is no more than the journal file Upload end position；The data renewal end position represents the daily record number of the last data renewal of the journal file According to the end of data position in the journal file.
The update mode of journal file involved by this programme can be diversified forms, for example, it may be incremental update, Can also be covering renewal or other forms.If journal file is by the way of covering updates, without the concern for daily record The data renewal end position of file, directly uploads journal file when uploading daily record data, it is not required that be considered as adopting Whether the journal file of collection object has been completed to gather, and is directly viewable whether journal file was once uploaded and the last data The time of renewal.In the present embodiment, for the purpose of simplifying the description, employ setting journal file using incremental update and every time The mode of new data is all added during renewal in end of file.
If as the journal file of an acquisition target in acquisition target list, last update time interval is preset Time point (such as when setting preset time point as specific monitoring collection object) exceedes given threshold, and journal file is most Nearly data renewal end position is no more than the upload end position of the journal file, then can make this acquisition target To have completed the acquisition target of collection.The recent renewal time of acquisition target can be from corresponding journal file attribute directly Obtain；Preset time point is exactly typically when being specifically monitored, naturally it is also possible to sets other rational time points；Set threshold Value depends on concrete condition, can specifically be set, for example, 1 day or 3 days.Recent renewal time exceedes away from preset time point Given threshold is the time limit that the recent renewal time of journal file has exceeded setting, i.e. renewal time has exceeded setting Time.The last end position information that uploads of acquisition target can obtain from uploading in end position information list；Data Renewal end position is no more than the last end position that uploads and shown, on this acquisition target (journal file), daily record number Finished according to all having uploaded.
Dynamic monitor component 207, for the time interval monitored according to the daily record, acquisition target described in regular monitoring arranges Table, and the acquisition target for having completed collection is removed from the acquisition target list, as periodically acquisition target is arranged Acquisition target in table is monitored, if specific acquisition target meets above-mentioned time conditions and data qualification simultaneously, so that it may To determine that this acquisition target to have completed to gather, when being updated to acquisition target list, can be removed from it this The acquisition target of collection is completed.For example, in the present embodiment, renewal time was completed into adopting for collection more than 1 day Collection object is deleted from acquisition target list.
Dynamic configuration component 205, is additionally operable to：Journal file set is added into the acquisition target list, by journal file Journal file in set is as acquisition target, and sets the time interval of the journal file set poll.
This programme not only can monitor and gather the individual log file as acquisition target, can also monitor and gather day Will file set.This programme can be monitored and gather to various forms of file sets, file set can be file, Listed files or the form of other any set.In the present embodiment, in order to simplify description, using file set as File illustrates.
Carry out the log data acquisition to journal file set, it is necessary first to it is configured by dynamic configuration component, it It is monitored by dynamic monitor component, is finally acquired by log collection component afterwards.First, dynamic configuration component is triggered, Acquisition target is configured in the dimension of file set.The journal file set for needing to be acquired is added into acquisition target List, while using the journal file in journal file set as acquisition target, and set journal file set poll when Between be spaced.
Afterwards, dynamic monitor component is triggered, according to the time interval of the journal file set poll, described in periodic polling Journal file set in acquisition target list；
If occurring newly-increased journal file in the journal file set, newly-increased journal file is set as acquisition target, And the acquisition target is added in the acquisition target list；
If the journal file in the journal file set is deleted, corresponding quilt is removed from the acquisition target list Delete the acquisition target of journal file.
, can be according to the time of the journal file set poll set by dynamic configuration component when dynamic monitor component is triggered Interval, the journal file set in periodic polling acquisition target list；Journal file set in poll acquisition target list Purpose is to look at the daily record for whether occurring in newly-increased journal file or this journal file set in this journal file set Whether file is deleted；If occurring newly-increased journal file in this journal file set, using newly-increased journal file as adopting Collect object, and the acquisition target of corresponding newly-increased journal file is added in acquisition target list；If this journal file set In journal file be deleted, then the acquisition target of corresponding deleted journal file is removed from acquisition target list.Pass through this The mode of sample periodic polling file set, dynamic monitor component realize the monitoring to journal file set.
Dynamic monitor component not only monitors the journal file among file set, but also monitors any among file set Journal file among the subclass of any level of journal file among the subclass of level, i.e. file set, is accordingly to be regarded as Journal file among file set is monitored and the processing as acquisition target addition into acquisition target list.
For example, in the present embodiment, there are entitled Sub file set, this preferred steps in acquisition target list This file set is monitored, the journal file quantity in this file set of periodic polling.In initial setting up, move State configuration component by this file set add acquisition target list, and using 10 journal files in this file set as Acquisition target.If dynamic monitor component finds there are 12 in entitled Sub file set in poll file set afterwards Journal file, then using newly increase 2 journal files as acquisition target, and newly increase 2 acquisition targets are added into collection List object；If found during poll file set, there was only 9 journal files in entitled Sub file set, then will be corresponding 1 acquisition target of the journal file newly deleted is deleted from acquisition target list.
The log collection component of this programme, it can trigger in for periodically whether being occurred according to acquisition target i.e. journal file Hold change, to determine how to gather specific daily record data.It can also change to any file content does not occur, i.e., not enter The journal file of any up-to-date of row carries out the collection of all or part of daily record data, can also be to there occurs file Content change, that is, the journal file for having carried out up-to-date carry out the collection of all or part of daily record data.
When specifically carrying out the collection to the daily record data of acquisition target, can trigger and journal file or and file set In the one-to-one log collection component of journal file in log collection sub-component；Corresponding one of each acquisition target is individually Log collection sub-component, it is ensured that it is non-interference between each log collection sub-component, there is provided Log Collect System it is steady It is qualitative；The daily record data that log collection sub-component can be newly increased with the acquisition target in taken at regular intervals list object, i.e., only enter Daily record data incremental crawler of the hand-manipulating of needle to the journal file as acquisition target, it is not necessary to each collection of repeated acquisition every time Object；Now, log collection sub-component needs to judge that what is the daily record data newly increased of acquisition target.
Due to the daily record data newly increased in this programme while including double implication, one kind is that do not have before in journal file The daily record data gathered is the daily record data newly increased；Another kind be in journal file before without upload daily record data be The daily record data newly increased.Because the daily record data of collection is all finally to gather the standard completed all to upload to be final, this Boundary standard is used as using upload end in embodiment, that is, obtains the last actual progress daily record data upload in journal file Position, to judge the upload performance of journal file, the daily record data newly increased not yet uploaded is also considered as what is newly increased Daily record data, carry out the collection of actual log data.
If the data renewal end position of journal file corresponding to the acquisition target exceedes last upload end position, Then to carrying out taken at regular intervals more than the last daily record data newly increased for uploading end position.
When the present embodiment specifically carries out the collection to the daily record data of acquisition target, it is first determined as acquisition target It is last in the data renewal end position and journal file of journal file to upload the position terminated；If data renewal terminates Position exceedes last upload end position, then shows the daily record data not yet uploaded be present, to terminating more than last time upload The data not yet uploaded of position are acquired.For example, in the present embodiment, the size of the current file of an acquisition target 1000KB, end of its data renewal end position for file is first confirmd that i.e. at 1000KB, and last time i.e. nearest one The secondary end position for completing to upload daily record data is 800KB, finds that the data renewal end position of journal file exceedes more afterwards Last time uploads end position, then the daily record data for showing to have 1000KB-800KB=200KB not yet uploads, can be from the last time Upload end position to rise, the 200KB daily record datas newly increased are acquired.
Fig. 6 is refer to, it is a kind of embodiment flow chart for log collection method that this programme 3rd embodiment provides.By Log collection unit embodiment is substantially similar in log collection method embodiment, so describe fairly simple, related part Referring to the explanation of log collection unit embodiment part.Log collection method embodiment described below is only schematic 's.The log collection method, main body is used corresponding to one, including：
Step S301：According to the demand using main body, for the acquisition target of setting, the acquisition target is gathered Daily record data；
Step S303：Upload gathered daily record data.
Optionally, it is described according to the demand using main body in execution, for the acquisition target of setting, adopted described in collection Before the step of collecting the daily record data of object, following step is performed：
Step S305：Using journal file as acquisition target, and the acquisition target is added into acquisition target list, and The time interval of daily record monitoring and the time interval of log collection are set；
Step S307：The time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from described The acquisition target for having completed collection is removed in acquisition target list.
One preferred steps of the present embodiment may be referred to Fig. 7.
Optionally, it is described to upload gathered daily record data；Including：
Limit the size that maximum single uploads data.
Optionally, it is described to upload gathered daily record data；Including：
Upload after completing, updated in end position information list is uploaded belonging to uploaded daily record data each time Acquisition target upload end position；The wherein described end position that uploads represents that the last daily record data for completing to upload exists End of data position in corresponding journal file.
Optionally, after completion is uploaded each time described in performing, for upper in end position information list is uploaded After the step of acquisition target belonging to the daily record data of biography carries out uploading the renewal of end position information, following step is performed：
The upload end position information list of renewal is stored on persistent storage medium.
Optionally, it is described according to the demand using main body, for the acquisition target of setting, gather the acquisition target Daily record data；Including：
According to the demand using main body, for the acquisition target in the acquisition target list, according to the daily record Whether the time interval of collection and the acquisition target change, taken at regular intervals daily record data.
Optionally, the acquisition target for having completed collection includes：
Journal file, the recent renewal time of the journal file exceed given threshold, and the day away from preset time point The data renewal end position of will file is no more than the upload end position of the journal file；
The data renewal end position represents the daily record data of the last data renewal of the journal file in institute State the end of data position in journal file.
Optionally, the setting journal file adds acquisition target list as acquisition target, and by the acquisition target, And the time interval of daily record monitoring and the time interval of log collection are set；Also include：
Journal file set is added into the acquisition target list, using the journal file in the journal file set as Acquisition target, and the time interval of the journal file set poll is set.
Optionally, the time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from institute State the acquisition target for being removed in acquisition target list and having completed collection；Also include：
According to the time interval of the journal file set poll, the daily record text in acquisition target list described in periodic polling Part set；
If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and The acquisition target is added in the acquisition target list；
If the journal file in the journal file set is deleted, removed from the acquisition target list with being deleted Except the corresponding acquisition target of journal file.
Optionally, it is described whether to be changed according to the time interval and the acquisition target of the log collection, periodically Gather daily record data；Including：
According to the time interval of the log collection, acquisition target newly increases in acquisition target list described in taken at regular intervals Daily record data.
Optionally, acquisition target newly increases daily record data in acquisition target list described in the taken at regular intervals, including：
If the data renewal end position of the journal file exceedes the upload end position of the journal file, to institute State the daily record data that newly increases uploaded between end position and data renewal end position and carry out taken at regular intervals.
Fig. 8 is refer to, it is a kind of embodiment schematic diagram for computing device that this programme fourth embodiment provides.Due to meter Calculate apparatus embodiments and be substantially similar to log collection method embodiment, so describing fairly simple, related part is referring to daily record Collecting unit and the explanation of log collection method embodiment part.Computing device embodiment described below is only schematic 's.
This programme provide a kind of computing device, be preconfigured to be it is associated with using main body, including：Processor 401 With memory 403；
The processor 401, for according to the demand using main body, for the acquisition target of setting, described in collection The daily record data of acquisition target；Upload gathered daily record data.
Although demand this programme is disclosed as above with preferred embodiment, it is not for limiting this programme, any ability Field technique personnel can make possible variation and modification, therefore this programme in the spirit and scope for not departing from this programme Protection domain should be defined by the scope that this programme claim is defined.
In a typical configuration, computing device includes one or more processors (CPU), input/output interface, net Network interface and internal memory.
Internal memory may include computer-readable medium in volatile memory, random access memory (RAM) and/or The forms such as Nonvolatile memory, such as read-only storage (ROM) or flash memory (flash RAM).Internal memory is computer-readable medium Example.
1st, computer-readable medium can be by any side including permanent and non-permanent, removable and non-removable media Method or technology realize that information stores.Information can be computer-readable instruction, data structure, the module of program or other numbers According to.The example of the storage medium of computer includes, but are not limited to phase transition internal memory (PRAM), static RAM (SRAM), dynamic random access memory (DRAM), other kinds of random access memory (RAM), read-only storage (ROM), Electrically Erasable Read Only Memory (EEPROM), fast flash memory bank or other memory techniques, read-only optical disc are read-only Memory (CD-ROM), digital versatile disc (DVD) or other optical storages, magnetic cassette tape, tape magnetic rigid disk storage or Other magnetic storage apparatus or any other non-transmission medium, the information that can be accessed by a computing device available for storage.According to Herein defines, and computer-readable medium does not include non-temporary computer readable media (transitory media), such as modulates Data-signal and carrier wave.
2nd, it will be understood by those skilled in the art that the embodiment of this programme can be provided as method, system or computer program production Product.Therefore, this programme can use the embodiment in terms of complete hardware embodiment, complete software embodiment or combination software and hardware Form.Moreover, this programme can use the computer for wherein including computer usable program code in one or more can use The computer program product that storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.) Form.

Claims (32)

  1. A kind of 1. Log Collect System, it is characterised in that including：
    Acquisition layer equipment, including more than one log collection unit；Wherein
    Each described log collection unit be preconfigured to be it is associated with using main body, and according to the need using main body Ask and gather daily record data, and the daily record data of collection is uploaded to accumulation layer equipment；
    The accumulation layer equipment, including more than one data storage cell；Wherein
    Each described data storage cell is corresponding with the log collection unit, and is adopted for storing corresponding daily record Collect the daily record data that unit uploads, and the daily record data of storage is supplied to the use master associated by the log collection unit Body.
  2. 2. Log Collect System according to claim 1, it is characterised in that the Log Collect System, be deployed in cloud system On system.
  3. 3. Log Collect System according to claim 1, it is characterised in that the daily record data, including：The industry of cloud system The daily record data for service process of being engaged in.
  4. 4. Log Collect System according to claim 1, it is characterised in that the acquisition layer equipment, in addition to：One with On monitoring unit；Wherein
    Each described monitoring unit is corresponding with the log collection unit, and by the log collection unit triggers, with And it is used for：Monitor whether the log collection unit has been triggered；
    If it is not, then trigger the log collection unit.
  5. 5. Log Collect System according to claim 4, it is characterised in that the log collection unit is additionally operable to：
    Monitor whether the monitoring unit has been triggered；
    If it is not, then trigger the monitoring unit.
  6. 6. Log Collect System according to claim 1, it is characterised in that the data storage cell, be additionally operable to：
    By using cloud storage service, the daily record data of upload is stored under the account item using main body.
  7. 7. Log Collect System according to claim 2, it is characterised in that the Log Collect System, be deployed in described On the separate unit service server of cloud system.
  8. 8. Log Collect System according to claim 2, it is characterised in that more than two Log Collect Systems, It is deployed in simultaneously on same service server of the cloud system.
  9. A kind of 9. log collection unit, it is characterised in that be preconfigured to be it is associated with using main body, including：Log collection Component, and daily record transmitting assembly；
    The log collection component, for according to the demand using main body, for the acquisition target of setting, being adopted described in collection Collect the daily record data of object；
    The daily record transmitting assembly, for uploading gathered daily record data.
  10. 10. log collection unit according to claim 9, it is characterised in that also include：Dynamic configuration component, and dynamic Monitor component；
    The dynamic configuration component, acquisition target is added for using journal file as acquisition target, and by the acquisition target List, and the time interval of daily record monitoring and the time interval of log collection are set；
    The dynamic monitor component, for the time interval monitored according to the daily record, acquisition target list described in regular monitoring, And the acquisition target for having completed collection is removed from the acquisition target list.
  11. 11. log collection unit according to claim 9, it is characterised in that the daily record transmitting assembly, be additionally operable to：
    Limit the size that maximum single uploads data.
  12. 12. log collection unit according to claim 10, it is characterised in that the daily record transmitting assembly, be additionally operable to：
    Upload each time after completing, adopting belonging to uploaded daily record data is updated in end position information list is uploaded Collect the upload end position of object；
    The wherein described end position that uploads represents number of the last daily record data for completing to upload in corresponding journal file According to end position.
  13. 13. log collection unit according to claim 12, it is characterised in that the daily record transmitting assembly, be additionally operable to：
    By the upload end position information list of renewal, persistent storage medium is stored in.
  14. 14. log collection unit according to claim 10, it is characterised in that the log collection component, including：
    At least one log collection sub-component, wherein
    Each described log collection sub-component is corresponding with the acquisition target in the acquisition target list, for according to institute The demand using main body is stated, whether changes according to the time interval and the acquisition target of the log collection, periodically adopts Collect daily record data.
  15. 15. log collection component according to claim 14, it is characterised in that be additionally operable to：
    According to the number change of acquisition target in the acquisition target list, dynamically increase and decrease the log collection sub-component.
  16. 16. log collection unit according to claim 12, it is characterised in that the acquisition target for having completed collection Including：
    Journal file, the recent renewal time of the journal file exceed given threshold, and daily record text away from preset time point The data renewal end position of part is no more than the upload end position of the journal file；
    The data renewal end position represents the daily record data of the last data renewal of the journal file in the day End of data position in will file.
  17. 17. log collection unit according to claim 10, it is characterised in that the dynamic configuration component, be additionally operable to：
    Journal file set is added into the acquisition target list, using the journal file in the journal file set as collection Object, and the time interval of the journal file set poll is set.
  18. 18. log collection unit according to claim 17, it is characterised in that the dynamic monitor component, be additionally operable to：
    According to the time interval of the journal file set poll, the journal file collection in acquisition target list described in periodic polling Close；
    If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and by institute Acquisition target is stated to add in the acquisition target list；
    If the journal file in the journal file set is deleted, removed and deleted day from the acquisition target list The corresponding acquisition target of will file.
  19. 19. log collection unit according to claim 17, it is characterised in that the log collection sub-component, be used for：It is fixed Acquisition target newly increases daily record data in the phase collection acquisition target list.
  20. 20. log collection unit according to claim 19, it is characterised in that the log collection sub-component, be used for：
    If the data renewal end position of the journal file exceedes the upload end position of the journal file, on described The daily record data that newly increases passed between end position and data renewal end position carries out taken at regular intervals.
  21. A kind of 21. log collection method, it is characterised in that main body is used corresponding to one, including：
    According to the demand using main body, for the acquisition target of setting, the daily record data of the acquisition target is gathered；
    Upload gathered daily record data.
  22. 22. log collection method according to claim 21, it is characterised in that described use main body according to described performing Demand, for the acquisition target of setting, before the step of gathering the daily record data of the acquisition target, perform following step：
    Using journal file as acquisition target, and the acquisition target is added into acquisition target list, and set daily record to monitor Time interval and log collection time interval；
    The time interval monitored according to the daily record, acquisition target list described in regular monitoring, and from the acquisition target list The acquisition target of collection has been completed in middle removal.
  23. 23. log collection method according to claim 21, it is characterised in that described to upload gathered daily record data； Including：
    Limit the size that maximum single uploads data.
  24. 24. log collection method according to claim 22, it is characterised in that described to upload gathered daily record data； Including：
    Upload each time after completing, adopting belonging to uploaded daily record data is updated in end position information list is uploaded Collect the upload end position of object；
    The wherein described end position that uploads represents number of the last daily record data for completing to upload in corresponding journal file According to end position.
  25. 25. log collection method according to claim 24, it is characterised in that uploading completion described in execution each time Afterwards, upload end position is carried out for the acquisition target belonging to the daily record data of upload in end position information list is uploaded After the step of renewal of information, following step is performed：
    The upload end position information list of renewal is stored on persistent storage medium.
  26. 26. log collection unit according to claim 22, it is characterised in that described according to the need using main body Ask, for the acquisition target of setting, gather the daily record data of the acquisition target；Including：
    According to the demand using main body, for the acquisition target in the acquisition target list, according to the log collection Time interval and the acquisition target whether change, taken at regular intervals daily record data.
  27. 27. log collection method according to claim 24, it is characterised in that the collection for having completed collection Object includes：
    Journal file, the recent renewal time of the journal file exceed given threshold, and daily record text away from preset time point The data renewal end position of part is no more than the upload end position of the journal file；
    The data renewal end position represents the daily record data of the last data renewal of the journal file in the day End of data position in will file.
  28. 28. log collection method according to claim 22, it is characterised in that the setting journal file is as collection pair As, and by the acquisition target add acquisition target list, and set daily record monitor time interval and log collection when Between be spaced；Also include：
    Journal file set is added into the acquisition target list, using the journal file in the journal file set as collection Object, and the time interval of the journal file set poll is set.
  29. 29. log collection method according to claim 28, it is characterised in that the time monitored according to the daily record Interval, acquisition target list described in regular monitoring, and removed from the acquisition target list and completed to adopt described in collection Collect object；Also include：
    According to the time interval of the journal file set poll, the journal file collection in acquisition target list described in periodic polling Close；
    If occurring newly-increased journal file in the journal file set, using newly-increased journal file as acquisition target, and by institute Acquisition target is stated to add in the acquisition target list；
    If the journal file in the journal file set is deleted, removed and deleted day from the acquisition target list The corresponding acquisition target of will file.
  30. 30. log collection method according to claim 28, it is characterised in that the time according to the log collection Whether interval and the acquisition target change, taken at regular intervals daily record data；Including：
    According to the time interval of the log collection, acquisition target newly increases daily record in acquisition target list described in taken at regular intervals Data.
  31. 31. log collection method according to claim 30, it is characterised in that acquisition target described in the taken at regular intervals arranges Acquisition target newly increases daily record data in table, including：
    If the data renewal end position of the journal file exceedes the upload end position of the journal file, on described The daily record data that newly increases passed between end position and data renewal end position carries out taken at regular intervals.
  32. 32. a kind of computing device, be preconfigured to be it is associated with using main body, including：Processor and memory；
    The processor, for according to the demand using main body, for the acquisition target of setting, gathering the acquisition target Daily record data；Upload gathered daily record data.
CN201610417661.3A 2016-06-14 2016-06-14 Log acquisition system, log acquisition unit and log acquisition method Active CN107508718B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610417661.3A CN107508718B (en) 2016-06-14 2016-06-14 Log acquisition system, log acquisition unit and log acquisition method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610417661.3A CN107508718B (en) 2016-06-14 2016-06-14 Log acquisition system, log acquisition unit and log acquisition method

Publications (2)

Publication Number Publication Date
CN107508718A true CN107508718A (en) 2017-12-22
CN107508718B CN107508718B (en) 2021-03-02

Family

ID=60679104

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610417661.3A Active CN107508718B (en) 2016-06-14 2016-06-14 Log acquisition system, log acquisition unit and log acquisition method

Country Status (1)

Country Link
CN (1) CN107508718B (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108170584A (en) * 2017-12-26 2018-06-15 广东欧珀移动通信有限公司 log processing method, device, storage medium and terminal device
CN109344139A (en) * 2018-11-01 2019-02-15 浪潮电子信息产业股份有限公司 A kind of polymerization and relevant apparatus of storage system operation log
CN110661631A (en) * 2018-06-28 2020-01-07 中兴通讯股份有限公司 Method, device and computer readable storage medium for collecting network element logs
CN111143161A (en) * 2019-12-09 2020-05-12 东软集团股份有限公司 Log file processing method and device, storage medium and electronic equipment
CN111723064A (en) * 2019-03-22 2020-09-29 顺丰科技有限公司 Log collection method and device, server and storage medium
CN112995307A (en) * 2021-02-09 2021-06-18 中国工商银行股份有限公司 Log processing method, system, device, electronic equipment and storage medium
CN113127306A (en) * 2021-04-23 2021-07-16 苏州若拙科技有限公司 Trusted log storage and verification system based on block chain
CN113687872A (en) * 2021-07-19 2021-11-23 北京鸿腾智能科技有限公司 Selection component control method, device, storage medium and apparatus
CN115576792A (en) * 2022-11-24 2023-01-06 北京宝兰德软件股份有限公司 Log collection system and method

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050198281A1 (en) * 2004-02-04 2005-09-08 Hon Hai Precision Industry Co., Ltd. System and method for logging events of network devices
CN101465765A (en) * 2008-12-31 2009-06-24 东信和平智能卡股份有限公司 Log system and use method thereof
US7599939B2 (en) * 2003-11-26 2009-10-06 Loglogic, Inc. System and method for storing raw log data
CN103944973A (en) * 2014-04-02 2014-07-23 北京中交兴路车联网科技有限公司 Method and device for collecting logs
CN102790686B (en) * 2011-05-17 2015-09-16 浙江核新同花顺网络信息股份有限公司 Log data acquisition method, system and server
CN104935469A (en) * 2015-06-30 2015-09-23 浪潮(北京)电子信息产业有限公司 Distributive storage method and system for log information

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7599939B2 (en) * 2003-11-26 2009-10-06 Loglogic, Inc. System and method for storing raw log data
US20050198281A1 (en) * 2004-02-04 2005-09-08 Hon Hai Precision Industry Co., Ltd. System and method for logging events of network devices
CN101465765A (en) * 2008-12-31 2009-06-24 东信和平智能卡股份有限公司 Log system and use method thereof
CN102790686B (en) * 2011-05-17 2015-09-16 浙江核新同花顺网络信息股份有限公司 Log data acquisition method, system and server
CN103944973A (en) * 2014-04-02 2014-07-23 北京中交兴路车联网科技有限公司 Method and device for collecting logs
CN104935469A (en) * 2015-06-30 2015-09-23 浪潮(北京)电子信息产业有限公司 Distributive storage method and system for log information

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108170584A (en) * 2017-12-26 2018-06-15 广东欧珀移动通信有限公司 log processing method, device, storage medium and terminal device
CN110661631A (en) * 2018-06-28 2020-01-07 中兴通讯股份有限公司 Method, device and computer readable storage medium for collecting network element logs
CN109344139A (en) * 2018-11-01 2019-02-15 浪潮电子信息产业股份有限公司 A kind of polymerization and relevant apparatus of storage system operation log
CN111723064A (en) * 2019-03-22 2020-09-29 顺丰科技有限公司 Log collection method and device, server and storage medium
CN111143161A (en) * 2019-12-09 2020-05-12 东软集团股份有限公司 Log file processing method and device, storage medium and electronic equipment
CN111143161B (en) * 2019-12-09 2024-04-09 东软集团股份有限公司 Log file processing method and device, storage medium and electronic equipment
CN112995307A (en) * 2021-02-09 2021-06-18 中国工商银行股份有限公司 Log processing method, system, device, electronic equipment and storage medium
CN112995307B (en) * 2021-02-09 2023-12-12 中国工商银行股份有限公司 Log processing method, system, device, electronic equipment and storage medium
CN113127306A (en) * 2021-04-23 2021-07-16 苏州若拙科技有限公司 Trusted log storage and verification system based on block chain
CN113687872A (en) * 2021-07-19 2021-11-23 北京鸿腾智能科技有限公司 Selection component control method, device, storage medium and apparatus
CN113687872B (en) * 2021-07-19 2024-03-29 三六零数字安全科技集团有限公司 Selection component control method, device, storage medium and apparatus
CN115576792A (en) * 2022-11-24 2023-01-06 北京宝兰德软件股份有限公司 Log collection system and method

Also Published As

Publication number Publication date
CN107508718B (en) 2021-03-02

Similar Documents

Publication Publication Date Title
CN107508718A (en) Log Collect System, log collection unit and log collection method
US7418489B2 (en) Method and apparatus for applying policies
US11593302B2 (en) Methods, systems, and computer readable mediums for implementing a data protection policy for a transferred enterprise application
CN106202346B (en) A kind of data load cleaning engine, scheduling and storage system
DE112011103666B4 (en) Storage management in cluster data processing systems
CN103677967B (en) A kind of remote date transmission system of data base and method for scheduling task
CN102438041B (en) Upgrade of highly available farm server groups
CN107567696A (en) The automatic extension of resource instances group in computing cluster
CN102523101B (en) Machine manager service fabric
CN105893628A (en) Real-time data collection system and method
CN102857371B (en) A kind of dynamic allocation management method towards group system
CN106202444A (en) A kind of implementation method of data base's O&M monitoring
CN109634716A (en) The OpenStack virtual machine High Availabitity management end device and management method of anti-fissure
CN107688611B (en) Saltstack-based Redis key value management system and method
CN110764871A (en) Cloud platform-based mimicry application packaging and control system and method
CN107547273A (en) A kind of support method and system of power system virtual instance High Availabitity
CN109614201A (en) The OpenStack virtual machine high-availability system of anti-fissure
CN108171050A (en) The fine granularity sandbox strategy method for digging of linux container
CN107908651A (en) A kind of auditing method of distributed type assemblies
DE112020003351T5 (en) Automatically detect ransomware with on-demand file system locking and an automatic repair feature
DE102016203598A1 (en) COMMUNITY COLLECTION OF DIAGNOSTIC DATA OF SOFTWARE PROGRAMS
CN105096014A (en) Method and system for recording work operation condition remotely
CN109981350A (en) A kind of continual upgrade method of cloud service and system
CN108228390A (en) Data return shelves method and device
CN105759247A (en) Radar comprehensive recorder and radar data recording method

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
REG Reference to a national code

Ref country code: HK

Ref legal event code: DE

Ref document number: 1248413

Country of ref document: HK

GR01 Patent grant
GR01 Patent grant