CN107508718B

CN107508718B - Log acquisition system, log acquisition unit and log acquisition method

Info

Publication number: CN107508718B
Application number: CN201610417661.3A
Authority: CN
Inventors: 宋军
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2016-06-14
Filing date: 2016-06-14
Publication date: 2021-03-02
Anticipated expiration: 2036-06-14
Also published as: CN107508718A

Abstract

This scheme provides a log collection system, includes: the acquisition layer equipment comprises more than one log acquisition unit; each log collection unit is configured to be associated with a use subject in advance, collect log data according to the requirements of the use subject, and upload the collected log data to the storage layer device; the storage layer device comprises more than one data storage unit; each data storage unit corresponds to the log acquisition unit and is used for storing the log data uploaded by the corresponding log acquisition unit and providing the stored log data to the use subject associated with the log acquisition unit. The scheme also provides a log acquisition unit and a log acquisition method. The distributed log collection system can solve the problems that a distributed log collection system in the prior art is lack of collection flexibility, high in system cost and complex in system deployment.

Description

Log acquisition system, log acquisition unit and log acquisition method

Technical Field

The scheme relates to the technical field of log data processing, in particular to a log acquisition system. The scheme also relates to a log acquisition unit and a log acquisition method.

Background

With the rapid development of cloud computing, a large number of proprietary cloud systems for specialized business scenes appear, and outstanding ones of the cloud systems include PaaS (platform as a service) and SaaS (software as a service). These cloud systems generally provide corresponding cloud services through a private cluster on the premise of using principal authorization; however, since the private cluster of these cloud systems is a black box for the using agent, the underlying information (e.g., virtual machines and storage) of the cloud service is invisible and uncontrollable to the using agent. In this case, it is important to be able to reflect the cluster operating state of the cloud system and the business process log of the relevant bottom layer information; using the subject requires the cloud system to provide accurate and real-time log data of the business process to assist it in checking whether the cloud service is safe and reliable.

At present, a plurality of distributed mass log acquisition systems are proposed in the prior art to meet the requirements of the log data acquisition, but in terms of system structure, the distributed log acquisition systems in the prior art are quite complex and generally divided into three layers:

a log collection layer comprising a plurality of log collection units;

a data collection layer comprising a plurality of data collection servers;

a data storage tier comprising a plurality of data storage servers.

Each log acquisition unit in the log acquisition layer is directly used for acquiring specific log data according to system presetting, and acquisition selection and adjustment cannot be made according to specific requirements of a final use main body of the log data; the fixed acquisition mode often causes the acquisition lack of pertinence and the waste of acquired data, and simultaneously requires that a data collection layer must be arranged to match with a corresponding data collection server to sort the log data acquired by the log acquisition unit and determine the attribution of the acquired log data relative to the using main body. And after the sorting and determining operations of the data collection layer are completed, the data collection layer is used for storing the collected data to the data storage layer.

In terms of cost, the distributed acquisition system in the prior art is divided into three structural layers in the longitudinal direction, and a data collection server is required to be deployed to sort log data and determine the attribution of the acquired log data; cannot be deployed on a single server in the horizontal direction, and a data collection server cluster and a data storage server cluster must be maintained. The deployment costs of such prior art distributed log collection systems are quite high.

In the practical application process, the amount of log data of a cloud system which is required by a main body is not large, deep analysis and mining on the log data are not performed, and only specific auxiliary and procedural log data are required. For example, when a big data processing cloud service is used, a user executes a specific job in a cloud system, and needs to check an output log of the specific job execution in time to ensure that the error reason can be checked when the specific job has an error. For the main body of the log data, the distributed log collection system in the prior art has the defects of poor collection flexibility, high cost of the whole set of system and complex system deployment.

Disclosure of Invention

The scheme provides a log acquisition system, a log acquisition unit and a log acquisition method, and aims to solve the problems that a distributed log acquisition system in the prior art is lack of acquisition flexibility, too high in system cost and too complex in system deployment.

This scheme provides a log collection system, includes:

the acquisition layer equipment comprises more than one log acquisition unit; wherein

Each log collection unit is configured to be associated with a use subject in advance, collect log data according to the requirements of the use subject and upload the collected log data to the storage layer equipment;

the storage layer device comprises more than one data storage unit; wherein

Each data storage unit corresponds to the log acquisition unit and is used for storing the log data uploaded by the corresponding log acquisition unit and providing the stored log data to the use body associated with the log acquisition unit.

Optionally, the log collection system is deployed on a cloud system.

Optionally, the log data includes: log data of a business service process of the cloud system.

Optionally, the acquisition layer apparatus further comprises: more than one monitoring unit; wherein

Each monitoring unit corresponds to the log collection unit, is triggered by the log collection unit, and is used for: monitoring whether the log collection unit has been triggered;

if not, triggering the log acquisition unit.

Optionally, the log collection unit is further configured to:

monitoring whether the monitoring unit has been triggered;

if not, the monitoring unit is triggered.

Optionally, the data storage unit is further configured to:

and storing the uploaded log data under the account item of the using subject by using a cloud storage service.

Optionally, the log collection system is deployed on a single service server of the cloud system.

Optionally, more than two log collection systems are deployed on the same service server of the cloud system at the same time.

The scheme also provides a log collection unit which is pre-configured to be associated with the use subject and comprises the following steps: the log acquisition component and the log uploading component;

the log acquisition component is used for acquiring log data of the acquisition object aiming at the set acquisition object according to the requirement of the using main body;

and the log uploading component is used for uploading the collected log data.

Optionally, the method further includes: a dynamic configuration component, and a dynamic monitoring component;

the dynamic configuration component is used for taking the log file as an acquisition object, adding the acquisition object into an acquisition object list, and setting a time interval for log monitoring and a time interval for log acquisition;

the dynamic monitoring component is used for regularly monitoring the collection object list according to the time interval of log monitoring and removing the collection objects which are collected from the collection object list.

Optionally, the log uploading component is further configured to:

the size of the maximum single upload data is defined.

Optionally, the log uploading component is further configured to:

after each uploading, updating the uploading ending position of the acquisition object to which the uploaded log data belongs in the uploading ending position information list;

and the uploading end position represents the data end position of the log data which is uploaded last time in the corresponding log file.

Optionally, the log uploading component is further configured to:

and storing the updated uploading end position information list in a persistent storage medium.

Optionally, the log collection component includes:

at least one log collection subcomponent wherein

Each log collection subassembly corresponds to a collection object in the collection object list and is used for collecting log data periodically according to the requirements of the use body and according to the time interval of log collection and whether the collection object changes or not.

Optionally, the method is further used for:

and dynamically increasing or decreasing the log collection subassemblies according to the change of the number of the collected objects in the collected object list.

Optionally, the acquisition object that has completed acquisition includes:

the updating method comprises the following steps that log files are stored, wherein the latest updating time interval of the log files exceeds a set threshold value from a preset time point, and the data updating end position of the log files does not exceed the uploading end position of the log files;

the data update ending position represents a data ending position of the log data of the latest data update of the log file in the log file.

Optionally, the dynamic configuration component is further configured to:

adding a log file set into the collection object list, taking the log files in the log file set as collection objects, and setting the polling time interval of the log file set.

Optionally, the dynamic monitoring component is further configured to:

periodically polling the log file set in the collection object list according to the polling time interval of the log file set;

if a new log file appears in the log file set, taking the new log file as an acquisition object, and adding the acquisition object into the acquisition object list;

and if the log files in the log file set are deleted, removing the acquisition objects corresponding to the deleted log files from the acquisition object list.

Optionally, the log collection subassembly is configured to: and acquiring newly-added log data of the acquisition objects in the acquisition object list periodically.

Optionally, the log collection subassembly is configured to:

and if the data updating end position of the log file exceeds the uploading end position of the log file, periodically collecting newly-added log data between the uploading end position and the data updating end position.

The scheme also provides a log collection method, which corresponds to a use main body and comprises the following steps:

according to the requirements of the user body, aiming at a set acquisition object, acquiring log data of the acquisition object;

and uploading the collected log data.

Optionally, before the step of acquiring log data of the set acquisition object according to the requirement of the user, the following steps are performed:

taking a log file as an acquisition object, adding the acquisition object into an acquisition object list, and setting a log monitoring time interval and a log acquisition time interval;

and monitoring the collection object list periodically according to the time interval of log monitoring, and removing the collection objects which are collected completely from the collection object list.

Optionally, the collected log data is uploaded; the method comprises the following steps:

the size of the maximum single upload data is defined.

Optionally, after each upload is completed, after the step of updating the upload end position information for the acquisition object to which the uploaded log data belongs in the upload end position information list, the following steps are performed:

and storing the updated uploading end position information list on a persistent storage medium.

Optionally, according to the requirement of the user, acquiring log data of a set acquisition object; the method comprises the following steps:

and according to the requirements of the using main body, aiming at the collection objects in the collection object list, regularly collecting log data according to the time interval of log collection and whether the collection objects change.

Optionally, the acquisition object that has completed acquisition includes:

Optionally, the log file is set as an acquisition object, the acquisition object is added into an acquisition object list, and a time interval for log monitoring and a time interval for log acquisition are set; further comprising:

Optionally, the collection object list is periodically monitored according to the time interval of log monitoring, and the collection objects which have been collected are removed from the collection object list; further comprising:

Optionally, the log data is periodically collected according to the time interval of log collection and whether the collection object changes; the method comprises the following steps:

and acquiring newly-added log data of the acquisition objects in the acquisition object list periodically according to the time interval of log acquisition.

Optionally, the periodically acquiring newly added log data of the acquisition objects in the acquisition object list includes:

The present solution also provides a computing device preconfigured to be associated with a usage principal, comprising: a processor and a memory;

the processor is used for acquiring log data of the acquisition object according to the requirement of the using main body and aiming at the set acquisition object; and uploading the collected log data.

Compared with the prior art, the scheme has the following advantages:

according to the scheme, the log acquisition unit is associated with the use main body, so that corresponding log data can be directly acquired aiming at the use main body, a data collection server is not required, and only two layers of simpler log acquisition and data storage are arranged on the longitudinal structure; meanwhile, according to the scheme, a complete log acquisition system can be formed by matching one log acquisition unit with one data storage unit, a bulky distributed cluster is not needed, and the log acquisition system can be simply and conveniently deployed on a single service server for use.

The log collection system provided by the scheme can flexibly configure the collection objects according to the specific requirements of the using main body, implements targeted log data collection, improves the efficiency of log data collection, and avoids waste caused by collecting redundant data.

The log collection system provided by the scheme has a simple structure, can be deployed on only a single service server, can also be deployed on any number of servers, and has good expansibility; meanwhile, the log collection system does not need the support of a server cluster, and the cost of system deployment is low.

The log data collected by the log collection system is specific to a specific using main body, the log data cannot be mixed with the log data of any other using main body, meanwhile, the log data are stored in the storage space specific to the specific using main body, and the data of the log collection system is high in safety and privacy.

The log acquisition system can also use independent cloud storage service to store log data, so that the structure of the log acquisition system is further simplified, the system deployment cost is reduced, and the security and privacy of the log data are also ensured.

Drawings

Fig. 1 is a schematic structural diagram of a distributed log collection system in the prior art provided by the present solution;

fig. 2 is a schematic structural diagram of a log collection system according to a first embodiment of the present disclosure;

fig. 3 is a schematic diagram of mutual monitoring between a log collection unit and a monitoring unit according to a first embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a log collection unit according to a second embodiment of the present disclosure;

fig. 5 is a schematic diagram of an upload end position information list according to a second embodiment of the present disclosure;

fig. 6 is a flowchart of a log collection method according to a third embodiment of the present disclosure;

FIG. 7 is a flowchart illustrating preferred steps of a log collection method according to a third embodiment of the present invention;

fig. 8 is a schematic diagram of a computing device according to a fourth embodiment of the present invention.

Detailed Description

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present solution. This solution can be implemented in many ways other than those described herein and those skilled in the art will appreciate that the present solution is not limited to the specific implementations disclosed below without departing from the scope of the present solution.

In the scheme, a log collection system, a log collection unit, a log collection method and a computing device are respectively provided. The following embodiments are described in detail with reference to the accompanying drawings.

Please refer to fig. 2, which is a schematic structural diagram of a log collection system according to a first embodiment of the present disclosure. The log collection system comprises:

the acquisition layer equipment 101 comprises more than one log acquisition unit 101-1; each log collection unit 101-1 is configured in advance to be associated with a use subject, collect log data according to the requirements of the use subject, and upload the collected log data to the storage layer device 103;

the storage layer device 103 comprises more than one data storage unit 103-1; each data storage unit 103-1 corresponds to the log collection unit 101-1, and is used for storing the log data uploaded by the corresponding log collection unit 101-1 and providing the stored log data to the use subject associated with the log collection unit 101-1.

The log acquisition system comprises acquisition layer equipment 101 with a plurality of log acquisition units and storage layer equipment 103 with a plurality of data storage units; each log collection unit is associated with a specific log data use main body in advance, collects log data according to the requirement of the specific log data use main body and uploads the collected log data to the data storage unit of the storage layer device 103; each data storage unit corresponds to a specific log collection unit and is specially used for storing the log data collected by the corresponding specific log collection unit and providing the stored log data to the use subject associated with the log collection unit.

The main body of use in the scheme is the user actually using the log data outside the log collection system, including the user served by the log collection system and other equipment or facilities needing to use the log data. In the scheme, each log acquisition unit is associated with a specific use main body; after the log collection unit is configured and associated with the specific use main body, the log collection unit is used exclusively by the specific use main body and cannot be configured and associated with other use main bodies. For example, in the present embodiment, as shown in fig. 2, the log collection unit 1, the log collection unit 2, and the log collection unit 3 each correspond to one use subject, that is, the log collection unit 1 corresponds to the use subject 1, the log collection unit 2 and the log collection unit 3 each correspond to the use subject 2, and the use subject 2 corresponds to two log collection units.

The log acquisition unit in the scheme is used for specifically implementing acquisition and uploading of log data. The log collection unit will be described in detail in a second embodiment of the present scheme.

The log data in the scheme can be log data of a business service process of the cloud system, namely the log data of the business process of the cloud system, such as calculation, processing, storage and the like, in the process of providing the business service of the main cloud system by the cloud system; the log data in this scheme may also be any other type of data required by the use subject, such as a specific service state of the service server. The log data in the present solution does not exclude the case that the application is deployed outside the cloud system, for example, the log data may also be the log data on a single service server.

The log collection system in the scheme can be a log collection system deployed on a cloud system. It can also be deployed in other distributed and non-distributed network systems besides cloud systems and even single server systems. In this embodiment, for simplicity of illustration, a log collection system deployed on a cloud system is preferred. In general, the use subject in the present scheme is also a use subject of the cloud system, and uses the service processing service provided by the cloud system.

Referring to fig. 1, the present solution is different from the prior art distributed log collection system in that:

first, each log collection unit that performs actual data collection in the collection device layer 101 of the present solution is configured to be associated with a usage subject (in one embodiment, one log collection unit may be associated with one usage subject, in another alternative embodiment, one log collection unit may be associated with multiple usage subjects, and in yet another alternative embodiment, multiple log collection units may be associated with one usage subject); the log collection unit of the log collection system in the prior art does not establish a corresponding relationship with any one of the using main bodies. The technical personnel in the field can understand that at least based on the clear association relationship established by each log acquisition unit and the using main body in the scheme, the log acquisition units can flexibly configure acquisition objects according to the specific requirements of the associated using main body on log data, thereby implementing targeted log data acquisition, improving the acquisition efficiency and avoiding acquiring unnecessary data; the log data acquisition of the log acquisition unit of the log acquisition system in the prior art is preset and fixed by the system, and the log data acquisition of the log acquisition unit cannot be flexibly customized according to the specific requirements of a specific using main body;

secondly, because each log collection unit of the scheme establishes a definite association relation with the use main body, the log collection unit can further establish a corresponding relation with the data storage unit of the use main body and can directly upload the collected data to the corresponding data storage unit by the log collection unit, therefore, the log collection system of the scheme omits a data collection layer which must be possessed in the prior art in the longitudinal structure, greatly simplifies the structure of the whole system and realizes the light system deployment effect. In contrast, the log collection unit in the prior art does not establish a corresponding relationship with any one of the use bodies (and the data storage unit for the use body), so a data collection layer must be deployed to sort the log data collected by the log collection unit and confirm which use body the collected data belongs to, which makes the structure of the whole system tend to be complex and increases the implementation cost;

thirdly, because each log acquisition unit of the scheme establishes association or corresponding relation with the use main body and/or the data storage unit, the log data acquired by the log acquisition unit is exclusive to the specific use main body and is directly uploaded to the data storage unit of the specific use main body by the log acquisition unit, so that the log data of the specific use main body cannot be mixed with the log data of other use main bodies in the whole process from acquisition to storage of the log data, and the safety and privacy of the log data can be ensured; in the log collecting system in the prior art, different log data (which may be attributed to different users) need to be sorted and confirmed via the data collecting layer during the collecting, collecting and storing processes, and therefore, the log data are mixed together, which increases the possibility that the log data is stolen, copied and maliciously forwarded, and is not beneficial to data privacy protection.

It should be noted that, because each log collection unit of the scheme establishes an association or a corresponding relationship with the use main body and/or the data storage unit, the flow of the collected log data in the log collection system of the scheme can only be longitudinal and not transverse, that is, the collected log data can only be uploaded to the data storage unit from the log collection unit and then provided to the use main body by the data storage unit. For example, the log data collected from the log collection unit 2 may only be uploaded to the data storage unit 2 and then supplied to the use subject 2. This log data is impossible to supply to the data storage unit 3.

As shown in fig. 3, in the present embodiment, the acquisition layer device 101 may include more than one monitoring unit 101-3 in addition to a plurality of log acquisition units; each monitoring unit 101-3 corresponds to the log collection unit 101-1 and is triggered by the log collection unit 101-1. And how many monitoring units can be correspondingly matched with how many log acquisition systems.

The monitoring unit is mainly used for monitoring whether the log acquisition unit 101-1 is triggered or not; if the log collection unit is not triggered, the log collection unit 101-1 is triggered. Furthermore, the log collection unit 101-1 may also monitor the monitoring unit 101-3 in reverse for monitoring whether the monitoring unit 101-3 has been triggered; if the monitoring unit 101-3 is not triggered, the monitoring unit 101-3 is triggered.

The monitoring of the log acquisition unit by the monitoring unit and the monitoring of the monitoring unit by the log acquisition unit can be real-time monitoring or periodic monitoring at intervals; the criteria for determining whether the log collection unit or the monitoring unit is not triggered can be flexibly determined. For example, the monitoring unit may periodically perform 1 monitoring on the log collection unit every 1 minute at set time intervals; whether the log collection unit is triggered can be judged according to the standard of whether the log collection unit is collecting or not.

The data storage unit 103-1 in the storage layer device 103 in the present embodiment corresponds to a specific log collection unit, that is, to a specific use subject. When the data storage unit is implemented in detail, a plurality of storage modes and storage media may be used to store the log data acquired by the corresponding log acquisition unit, for example, the log data is directly stored on a hard disk on a single server, or stored in a network hard disk in a local area network, or stored under a cloud storage account provided by the same cloud system deployed by the log acquisition system, and the like. The using main body must be able to obtain the log data stored in the data storage unit, no matter what specific way is adopted, such as directly reading from a local server hard disk or a local area network hard disk, or downloading from a network, etc.

The data storage unit 103-1 in the storage tier device 103 in this embodiment is further configured to store the uploaded log data in the account item of the user by using a cloud storage service.

When the cloud storage service providing the independent storage account is used for storing data, all log data of one using subject can be stored under the cloud storage account item belonging to the using subject. At the moment, on one hand, the independence and the privacy among the accounts of the cloud storage system can be utilized to ensure the safety of the stored log data; on the other hand, cloud storage service except the log acquisition system can be utilized to further simplify the structure of the log acquisition system; the method is substantially equivalent to outsourcing the storage layer equipment of the log collection system, and the cloud storage service is utilized to implement the function of the storage layer equipment of the log collection system, so that the whole log collection system is lightened. Of course, in one embodiment, log data of a using subject may be stored in one or more cloud storage accounts of a cloud storage system to which the using subject belongs; in another alternative embodiment, log data of one usage subject may also be stored in a plurality of accounts of a plurality of cloud storage systems. The embodiment preferably uses the cloud storage service provided by the cloud system itself in which the log collection system is deployed, so that the efficiency of storing data is high. For example, in this embodiment, when the main body 1 has an independent cloud storage authority in a cloud system deployed by the log collection system, the corresponding data storage unit may store log data uploaded by the corresponding log collection unit under the cloud storage account item of the main body 1 by using the cloud storage account and the password of the main body 1.

The log collection system in the scheme can be transversely expanded. According to the specific requirements of log data acquisition, a plurality of log acquisition systems corresponding to one or more use subjects can be deployed on a plurality of service servers of the cloud system, and the horizontal extension is almost unlimited; the log collection system in the scheme can also be transversely simplified. According to the specific requirements of log data acquisition, a minimum log acquisition system corresponding to a using main body, namely a log acquisition unit and a data storage unit, can be directly deployed on a single service server of a cloud system; or more than two log acquisition units corresponding to the two using bodies are deployed on the same service server of the cloud system. For example, in this embodiment, the log collecting system in fig. 2 is deployed on a single service server, where the log collecting unit 1 and the log collecting units 2 and 3 collect log data respectively by using the main body 1 and the main body 2.

Please refer to fig. 4, which is a schematic structural diagram of a log collection unit according to a second embodiment of the present disclosure. The log collection unit 101 of the present scheme includes: a log collection component 201 and a log uploading component 203;

the log collection component 201 is configured to collect log data of a set collection object according to the requirement of the user.

The log collection component 201 is configured in the log collection unit 101, and is configured to implement, according to a requirement of a usage subject corresponding to the log collection unit for the log files, collection of specific log data for the log files that are collection targets.

The log uploading component 203 is configured to upload the collected log data.

And the log uploading component 203 is used for uploading the log data collected by the log collecting component 201 in the log collecting unit 101.

Therefore, a minimum log collection unit can comprise a log collection component and a log uploading component; if a data storage unit is added, a minimum log collection system can be formed.

The log collection unit of the scheme further comprises: a dynamic configuration component 205, and a dynamic monitoring component 207.

The dynamic configuration component 205 is configured to use the log file as a collection object, add the collection object to a collection object list, and set a time interval for log monitoring and a time interval for log collection.

The dynamic configuration component 205 in the log collection unit of the present scheme is configured to use the log file as a collection object, add the collection object into the collection object list, that is, add a newly set collection object into the collection object list, and set a time interval for log monitoring and a time interval for log collection. Basic parameters required for log collection can be set by the present component 205, and collection activities can be monitored and managed through these set basic parameters.

The collection object of the scheme can be in various forms, and can be a log file of a business service process of the cloud system, namely the log file of the business process of the cloud system, which is particularly related to calculation, processing, storage and the like, is recorded in the process of providing the business service using the main cloud system; the collection object in the scheme may also be any other type of data required by the use subject, such as a specific service state of the service server. The collection object in the scheme does not exclude the situation that the application is deployed outside the cloud system, for example, the collection object can also be a log file on a single service server. In this embodiment, to simplify the description, the acquisition objects correspond to specific log files, that is, each acquisition object is a log file added to the acquisition object list; the log file is used as a collection object, namely the log file is confirmed to be in a state to be collected, no change or mark is made on the log file, and the log file is only classified into a category to be collected. In particular implementation, a file name wildcard may be set to support a log file satisfying the name wildcard as a collection object.

The collection object list comprises collection objects collected by the log collection unit, namely log files of states to be collected, the log monitoring component determines specific collection objects monitored by the log monitoring component according to the contents of the collection object list, and the log collection component also determines the specific log files collected by the log collection component according to the collection object list. For example, 20 log files are added to the collection object list as collection objects, that is, the 20 log files are confirmed to be in a state to be collected.

The time interval of log monitoring and the time interval of log collection are both time periods of specific duration set automatically or manually by the system, and are generally in units of seconds. The two time intervals are both prepared for the subsequent log monitoring step and the log collecting step; the shorter the time interval, the more real-time the log monitoring and log collection. For example, the time interval for log monitoring is set to 30 seconds, and the time interval for log collection is set to 10 seconds.

The dynamic monitoring component 207 is configured to periodically monitor the collection object list according to the time interval of log monitoring, and remove a collection object that has completed collection from the collection object list.

The dynamic monitoring component 207 is configured to periodically monitor the collection object list set by the dynamic configuration component 205 according to the time interval of log monitoring set by the dynamic configuration component 205, and remove the collection object that has completed collection from the collection object list; the dynamic monitoring component 207 is further configured to specifically monitor the collection object list, and remove and complete the collected collection objects, so as to prevent the collection objects from increasing and exploding the collection object list.

The dynamic monitoring component 207 is configured to trigger the dynamic monitoring component 207 periodically according to the time interval of log monitoring set by the dynamic configuration component 205, that is, at every set time interval of log monitoring, and so on.

After the dynamic configuration component 205 sets specific parameters required for log collection and the dynamic monitoring component 207 monitors the collection object list, the log collection component 201 of the present scheme may be further refined. The log collection component 201 further comprises a plurality of log collection sub-components 201-1, wherein each log collection sub-component corresponds to a collection object in the collection object list and is used for collecting log data periodically according to the requirement of the user body and according to the time interval of log collection and whether the collection object changes.

The log collection component 201 includes one or more log collection subcomponents 201-1 corresponding to a single collection object; each log collection subassembly corresponds to a collection object in the collection object list one by one, and how many log collection subassemblies are correspondingly triggered when how many collection objects are in the collection object list; each log collection subassembly is used for regularly collecting corresponding log data according to the specific requirements of the use main body corresponding to the log collection unit and according to the set time interval of log collection and whether the collection object changes on the file level. For example, in this embodiment, as shown in fig. 3, the log collection component includes 3 log collection sub-components, which respectively correspond to 3 log files serving as collection objects to collect log data.

The log collection component 201 further comprises: and dynamically increasing or decreasing the log collection subassemblies according to the change of the number of the collected objects in the collected object list. If the number of the acquisition objects in the acquisition object list is increased, the log acquisition assembly dynamically adjusts the number of the log acquisition subassemblies, and correspondingly increases the log acquisition subassemblies so that the increased log acquisition subassemblies correspond to the increased acquisition objects; if the number of the collection objects in the collection object list is reduced, the log collection component dynamically adjusts the number of the log collection sub-components, reduces the corresponding log collection sub-components, and correspondingly removes the log collection sub-components corresponding to the collection objects removed from the collection object list. The number of log collection subcomponents is dynamically changed as the number of collection objects in the collection object list changes.

The log uploading component 203 of the present scheme is further configured to: the size of the maximum single upload data is defined.

The log uploading component may comply with a limit of the uploading bandwidth when uploading the log data collected by the log collecting component, that is, the maximum single uploading data size does not exceed the limited bandwidth upper limit. The limit of the uploading bandwidth is set to prevent the log acquisition system from causing network traffic blockage of the deployed service server, and the stability of the system operation is ensured.

For example, if the size of the maximum single upload data is defined as 500KB, and it is assumed that the log collection unit newly collects 600KB log data for a certain collection object corresponding to a certain usage body, 500KB log data can be uploaded only once when the log upload component is triggered.

A log uploading component 203, further configured to: after each uploading, updating the uploading ending position of the acquisition object to which the uploaded log data belongs in the uploading ending position information list; and the uploading end position represents the data end position of the log data which is uploaded last time in the corresponding log file.

When the log uploading component 203 is configured to upload the log data acquired by the log acquiring component, the uploading end position may be updated in the uploading end position information list after each uploading is completed, with respect to the acquisition object to which the uploaded log data belongs.

The uploading end position of the collection object is the data position of the log data which is uploaded for the last time in the log file as the specific collection object, and the uploading end position can determine the position limit of the log data which is uploaded and not uploaded in the log file. For example, the log file size of one collection object is 1000 KB. In one embodiment, when the data end position where the log data is uploaded last time is determined to be 800KB, it may be determined that 800KB log data has been uploaded and 1000KB-800 KB-200 KB log data has not been uploaded in the log file.

The upload end position information list is a general table listing the upload end position information of the collection objects, and specifies the end position of the latest upload data of each collection object in the log file. The specific form of the upload end position information list is a one-to-one two-dimensional table, please refer to fig. 5, which is a content diagram of an upload end position information list according to a second embodiment of the present disclosure. One column in the upload end position information list is a log file as a collection object, and the other column is a recorded specific position where the last upload of the corresponding log file is ended.

Because other components except the log uploading unit need to acquire or update the uploading end position information of the acquisition object, after the log uploading component is triggered each time to finish uploading the log data, the log uploading component can also be used for updating the uploading end position information of the corresponding acquisition object in the uploading end position information list for the acquisition object to which the log data belongs.

For example, in the present embodiment, a log file as a collection target has a size of 1000KB, and the end position of the last time, that is, the latest time of completing uploading the log data is 800KB, that is, in the upload end position information list, the upload end position is recorded as 800 KB. After the collection of the 200KB log data is continued from the 800KB position, and accordingly, the 150KB log data in the 200KB log data is again uploaded, the upload end position information list is updated to 950KB for the collection object having the file size of 1000 KB.

After the log uploading component is triggered to complete updating the uploading end position information list, the log uploading component can also be used for storing the updated uploading end position information list in a persistent storage medium.

By persistently storing the updated uploading end position information list, for example, in a related storage directory of a local hard disk, the uploading end position information loss caused by sudden power failure and downtime of a service server can be prevented, so that repeated log data uploading is avoided, and the stability and efficiency of the log acquisition system are improved.

The acquisition object for which acquisition has been completed includes: the updating method comprises the following steps that log files are stored, wherein the latest updating time interval of the log files exceeds a set threshold value from a preset time point, and the data updating end position of the log files does not exceed the uploading end position of the log files; the data update ending position represents a data ending position of the log data of the latest data update of the log file in the log file.

The updating mode of the log file related to the scheme can be in various forms, for example, the updating mode can be incremental updating, or coverage updating or other forms. If the log file adopts a coverage updating mode, the data updating end position of the log file is not required to be considered, the log file is directly uploaded when the log data is uploaded, whether the log file serving as a collection object is collected or not is not required to be considered, and whether the log file is uploaded once or not and the time of the latest data updating are directly checked. In this embodiment, for simplicity of description, a mode is adopted in which the log file is set to be updated incrementally and new data is added to the end of the file each time the log file is updated.

If the log file as one collection object in the collection object list has a last update time interval from a preset time point (for example, when the preset time point is set as a specific monitoring collection object) exceeding a set threshold, and the last data update end position of the log file does not exceed the upload end position of the log file, this collection object may be used as a collection object for which collection has been completed. The latest updating time of the collection object can be directly obtained from the corresponding log file attribute; the preset time point is generally when monitoring is specifically performed, and other reasonable time points can be set; the threshold value is set depending on the circumstances, and may be set specifically, for example, 1 day or 3 days. If the latest updating time is more than the preset threshold from the preset time point, the latest updating time of the log file exceeds the set time limit, namely the updating time exceeds the set time. The latest uploading end position information of the acquisition object can be obtained from the uploading end position information list; the data updating end position does not exceed the last uploading end position, which indicates that all log data are uploaded on the collection object (log file).

The dynamic monitoring component 207 is configured to periodically monitor the collection object list according to the time interval of log monitoring, and remove the collection object that has completed collection from the collection object list, that is, periodically monitor the collection objects in the collection object list, if a specific collection object meets the time condition and the data condition at the same time, it may be determined that the collection object has completed collection, and when the collection object list is updated, the collection object that has completed collection may be deleted therefrom. For example, in the present embodiment, the acquisition objects whose acquisition has been completed with the update time exceeding 1 day are deleted from the acquisition object list.

A dynamic configuration component 205 further operable to: adding the log file set into the collection object list, taking the log files in the log file set as collection objects, and setting the polling time interval of the log file set.

The scheme can monitor and collect not only a single log file as a collection object, but also a log file set. The scheme can monitor and collect file sets in different forms, and the file sets can be folders, file lists or any other set form. In the present embodiment, for the sake of simplifying the description, a file set is explained as a folder.

The log data of the log file set is acquired by firstly setting through the dynamic configuration component, then monitoring through the dynamic monitoring component, and finally acquiring through the log acquisition component. Firstly, a dynamic configuration component is triggered, and a collection object is set on the dimension of a file set. Adding a log file set needing to be collected into a collection object list, taking a log file in the log file set as a collection object, and setting a polling time interval of the log file set.

Then, triggering a dynamic monitoring component, and periodically polling the log file set in the collection object list according to the time interval of the polling of the log file set;

if a newly added log file appears in the log file set, setting the newly added log file as an acquisition object, and adding the acquisition object into the acquisition object list;

When the dynamic monitoring component is triggered, the log file set in the collection object list is periodically polled according to the time interval of the polling of the log file set by the dynamic configuration component; the purpose of polling the log file set in the collection object list is to check whether a newly added log file appears in the log file set or whether the log file in the log file set is deleted; if a new log file appears in the log file set, taking the new log file as an acquisition object, and adding the acquisition object corresponding to the new log file into an acquisition object list; if the log file in the log file set is deleted, removing the collection object corresponding to the deleted log file from the collection object list. By means of the mode of periodically polling the file set, the dynamic monitoring component realizes monitoring of the log file set.

The dynamic monitoring component monitors not only the log files in the file set, but also the log files in any hierarchy subset in the file set, namely the log files in any hierarchy subset in the file set, and all the log files in the file set are regarded as the log files in the file set to be monitored and added into the collection object list as collection objects.

For example, in this embodiment, there is a file set named Sub in the collection object list, and this preferred step is to monitor this file set, and periodically poll the number of log files in this file set. At initial setup, the dynamic configuration component adds this set of files to the collection object list and takes 10 log files in this set of files as collection objects. If the dynamic monitoring component finds that 12 log files exist in the file set named Sub when polling the file set, taking the newly added 2 log files as collection objects, and adding the newly added 2 collection objects into a collection object list; if only 9 log files are found in the file set named Sub when the file set is polled, 1 collection object corresponding to the newly deleted log file is deleted from the collection object list.

The log collection component can be triggered to determine how to collect specific log data according to whether the content of a collection object, namely a log file changes or not. The method can also collect all or part of log data of the log file which has not changed the file content, that is, has not updated the file content, or collect all or part of log data of the log file which has changed the file content, that is, has updated the file content.

When the log data of the collected object is collected, the log collection subassemblies in the log collection assemblies corresponding to the log files or the log files in the file set one by one can be triggered; each acquisition object corresponds to an independent log acquisition subassembly, so that the log acquisition subassemblies are ensured not to interfere with each other, and the stability of a log acquisition system is improved; the log collection subassembly can also regularly collect the newly added log data of the collection objects in the object list, namely only performing the incremental collection of the log data of the log file as the collection object without repeatedly collecting each collection object each time; at this time, the log collection subcomponent needs to determine what is the newly added log data of the collection object.

Because the newly added log data in the scheme simultaneously comprises two meanings, one is that the log data which is not collected before in the log file is the newly added log data; the other is that the log data which is not uploaded before in the log file is newly added log data. In the embodiment, the uploading end is used as a boundary standard, that is, the last actual log data uploading position in the log file is obtained to judge the uploading completion condition of the log file, and newly-added log data which is not uploaded yet is also regarded as newly-added log data to collect the actual log data.

And if the data updating end position of the log file corresponding to the acquisition object exceeds the last uploading end position, periodically acquiring newly-added log data exceeding the last uploading end position.

When the embodiment specifically collects the log data of the collection object, first, a data update end position of the log file as the collection object and a position where the last upload of the log file is ended are determined; if the data updating end position exceeds the last uploading end position, the log data which are not uploaded exist, and the data which exceed the last uploading end position and are not uploaded are collected. For example, in this embodiment, the size of the current file of a collection object is 1000KB, it is first confirmed that the data update end position is 1000KB, which is the end of the file, and the end position of the last time that the uploading of log data is completed is 800KB, and after comparison, it is found that the data update end position of the log file exceeds the last upload end position, which indicates that log data of 1000KB-800 KB-200 KB has not been uploaded, and the newly added 200KB log data can be collected from the last upload end position.

Please refer to fig. 6, which is a flowchart illustrating an embodiment of a log collecting method according to a third embodiment of the present disclosure. Because the embodiment of the log collection method is basically similar to the embodiment of the log collection unit, the description is simple, and the relevant points can be referred to the description of the embodiment part of the log collection unit. The log collection method embodiments described below are merely illustrative. The log collection method corresponds to a use subject and comprises the following steps:

step S301: according to the requirements of the user body, aiming at a set acquisition object, acquiring log data of the acquisition object;

step S303: and uploading the collected log data.

step S305: taking a log file as an acquisition object, adding the acquisition object into an acquisition object list, and setting a log monitoring time interval and a log acquisition time interval;

step S307: and monitoring the collection object list periodically according to the time interval of log monitoring, and removing the collection objects which are collected completely from the collection object list.

A preferred step of this embodiment can be seen in fig. 7.

the size of the maximum single upload data is defined.

after each uploading, updating the uploading ending position of the acquisition object to which the uploaded log data belongs in the uploading ending position information list; and the uploading end position represents the data end position of the log data which is uploaded last time in the corresponding log file.

Optionally, the acquisition object that has completed acquisition includes:

Please refer to fig. 8, which is a schematic diagram illustrating an embodiment of a computing device according to a fourth embodiment of the present disclosure. Because the embodiment of the computing device is basically similar to the embodiment of the log collection method, the description is simple, and the relevant points can be referred to the description of the log collection unit and the embodiment of the log collection method. The computing device embodiments described below are merely illustrative.

The scheme provides a computing device which is pre-configured to be associated with a usage subject and comprises the following components: a processor 401 and a memory 403;

the processor 401 is configured to collect log data of a set collection object according to a requirement of the user body and for the collection object; and uploading the collected log data.

It is to be understood that the present invention is not limited to the above embodiments, but may be modified and adapted by those skilled in the art without departing from the spirit and scope of the present invention.

In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.

1. Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.

2. As will be appreciated by one skilled in the art, embodiments of the present solution may be provided as a method, system, or computer program product. Accordingly, the present solution may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present solution may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

Claims

1. A log collection system, comprising:

the storage layer device comprises more than one data storage unit; wherein

2. The log collection system of claim 1, wherein the log collection system is deployed on a cloud system.

3. The log collection system of claim 1, wherein the log data comprises: log data of a business service process of the cloud system.

4. The log collection system of claim 1, wherein the collection layer device further comprises: more than one monitoring unit; wherein

if not, triggering the log acquisition unit.

5. The log collection system of claim 4, wherein the log collection unit is further configured to:

monitoring whether the monitoring unit has been triggered;

if not, the monitoring unit is triggered.

6. The log collection system of claim 1, wherein the data storage unit is further configured to:

7. The log collection system of claim 2, wherein the log collection system is deployed on a single service server of the cloud system.

8. The log collection system of claim 2, wherein more than two log collection systems are deployed on the same service server of the cloud system at the same time.

9. A log collection unit, preconfigured to be associated with a usage principal, comprising: the log acquisition component and the log uploading component;

and the log uploading component is used for uploading the collected log data.

10. The log collection unit of claim 9, further comprising: a dynamic configuration component, and a dynamic monitoring component;

11. The log collection unit of claim 9, wherein the log uploading component is further configured to:

the size of the maximum single upload data is defined.

12. The log collection unit of claim 10, wherein the log uploading component is further configured to:

13. The log collection unit of claim 12, wherein the log uploading component is further configured to:

14. The log collection unit of claim 10, wherein the log collection component comprises:

at least one log collection subcomponent wherein

15. The log collection component of claim 14, further configured to:

16. The log collecting unit according to claim 12, wherein the collection object whose collection has been completed includes:

17. The log collection unit of claim 10, wherein the dynamic configuration component is further configured to:

18. The log collection unit of claim 17, wherein the dynamic monitoring component is further configured to:

19. The log collection unit of claim 17, wherein the log collection subcomponent is configured to: and acquiring newly-added log data of the acquisition objects in the acquisition object list periodically.

20. The log collection unit of claim 19, wherein the log collection subassembly is to:

21. A log collection method, corresponding to a usage subject, comprising:

and uploading the collected log data.

22. The log collecting method according to claim 21, wherein before performing the step of collecting log data of the collection object for the set collection object according to the demand of the use subject, the following steps are performed:

23. The log collection method of claim 21, wherein uploading the collected log data; the method comprises the following steps:

the size of the maximum single upload data is defined.

24. The log collection method of claim 22, wherein said uploading the collected log data; the method comprises the following steps:

25. The log collecting method according to claim 24, wherein after each uploading is completed, the following steps are performed after the step of updating the uploading end position information for the collection object to which the uploaded log data belongs in the uploading end position information list:

26. The log collection unit according to claim 22, wherein the log data of the collection object is collected for a set collection object according to a requirement of the user; the method comprises the following steps:

27. The log collecting method as claimed in claim 24, wherein the collection object whose collection has been completed includes:

28. The log collecting method as claimed in claim 22, wherein the log file is set as a collecting object, and the collecting object is added to a collecting object list, and a time interval of log monitoring and a time interval of log collecting are set; further comprising:

29. The log collecting method as claimed in claim 28, wherein said collecting object list is monitored periodically at time intervals of said log monitoring, and said collecting object whose collection has been completed is removed from said collecting object list; further comprising:

30. The log collecting method as claimed in claim 28, wherein the log data is collected periodically according to a time interval of the log collection and whether the collection object is changed; the method comprises the following steps:

31. The log collection method of claim 30, wherein said periodically collecting new log data for collection objects in said collection object list comprises:

32. A computing device preconfigured to be associated with a usage principal, comprising: a processor and a memory;