CN110011845B

CN110011845B - Log collection method and system

Info

Publication number: CN110011845B
Application number: CN201910255935.7A
Authority: CN
Inventors: 张凯顺
Original assignee: New H3C Big Data Technologies Co Ltd
Current assignee: New H3C Big Data Technologies Co Ltd
Priority date: 2019-03-29
Filing date: 2019-03-29
Publication date: 2022-05-10
Anticipated expiration: 2039-03-29
Also published as: CN110011845A

Abstract

The invention provides a log collection method and a log collection system, and relates to the technical field of communication. The log collection method comprises the following steps: the plurality of acquisition components acquire a plurality of log files, the plurality of acquisition components store the plurality of log files to the plurality of message queues, the plurality of analysis components analyze the log files in the plurality of message queues to obtain a plurality of analyzed log files, and the plurality of analysis components store the plurality of analyzed log files to the database. The log file acquisition system has the advantages that the log files are acquired through the acquisition assemblies in the log acquisition system, and the log files are analyzed through the analysis assemblies, so that the condition that the log files cannot be acquired and analyzed when any one acquisition assembly or analysis assembly is abnormal is avoided, the coupling degree in the process of acquiring the log files is reduced, and the reliability and flexibility of acquiring the log files are improved.

Description

Log collection method and system

Technical Field

The invention relates to the technical field of communication, in particular to a log collection method and a log collection system.

Background

With the continuous development of the internet and big data, more and more logs are generated by network equipment, and a user can analyze the big data based on the logs to obtain valuable data in the logs.

In the related art, the log files generated by each network device can be collected through the collection component, the collected log files are analyzed through the analysis component, and finally the analyzed data are stored in a preset data storage system.

However, the log collection process is linear correlation, the coupling degree is high, and when the collection assembly, the analysis assembly or the data storage system is abnormal, the log cannot be normally collected and stored.

Disclosure of Invention

The present invention aims to provide a log collecting method and device to solve the problems that the coupling degree of the collected logs is high, and the logs cannot be normally collected and stored when the collection assembly, the analysis assembly or the data storage system is abnormal.

In order to achieve the above purpose, the embodiment of the present invention adopts the following technical solutions:

in a first aspect, an embodiment of the present invention provides a log collecting method, which is applied to a log collecting system, where the log collecting system includes: the system comprises a plurality of acquisition components, a plurality of analysis components, a plurality of message queues and a database, and the method comprises the following steps:

a plurality of collecting components collect a plurality of log files;

the plurality of collecting components store a plurality of log files to a plurality of message queues;

the plurality of analysis components analyze the log files in the plurality of message queues to obtain a plurality of analyzed log files;

and the plurality of analysis components store the plurality of analyzed log files to the database.

Optionally, the collecting components collect log files, including:

monitoring the collection state of each log file according to a cache directory;

and for each log file, if the acquisition state indicates that the log file is in a state to be acquired, acquiring the log file through any idle acquisition assembly in the plurality of acquisition assemblies.

Optionally, the method further includes:

if any log file in the plurality of log files is detected to be completely acquired, updating the acquisition state corresponding to the log file which is completely acquired in the cache directory to be completely acquired;

and if the acquisition states corresponding to the plurality of log files in the cache directory are detected to be all acquired, deleting the cache directory.

Optionally, the collecting the log file by any idle collecting component in the plurality of collecting components includes:

if the first idle acquisition assembly in the plurality of acquisition assemblies is in an abnormal state in the process of acquiring the log file, generating abnormal position information;

and a second idle acquisition assembly in the plurality of acquisition assemblies continues to acquire the log file according to the abnormal position information.

Optionally, the generating of the abnormal position information includes:

acquiring an interruption position, wherein the interruption position is used for indicating a position at which the first idle acquisition assembly interrupts the acquisition of the log file;

and generating the abnormal position information according to the interrupt position.

Optionally, the storing, by the plurality of collecting components, the plurality of log files to the plurality of message queues includes:

the plurality of collecting assemblies store the plurality of log files in a cache queue according to the collecting time corresponding to each log file, and the collecting time is used for indicating the time corresponding to the collected log files; wherein any one of the log files in the buffer queue is randomly stored to one of the plurality of message queues.

Optionally, the parsing the log files in the message queues by the parsing components to obtain a plurality of parsed log files includes:

monitoring the working states of a plurality of analysis components;

and for each analysis assembly, if the analysis assembly is detected to be in an idle state, randomly reading one log file in one message queue through the idle analysis assembly to analyze, and obtaining the analyzed log file.

Optionally, the storing, by the plurality of parsing components, the plurality of parsed log files to the database includes:

and the plurality of analysis components store each analyzed log file into a storage space corresponding to the log identifier in the database according to the log identifier corresponding to the plurality of analyzed log files.

In a second aspect, an embodiment of the present invention further provides a log collecting system, where the log collecting system includes: the system comprises a plurality of acquisition components, a plurality of message queues, a plurality of analysis components and a database;

the plurality of acquisition components are used for acquiring a plurality of log files;

the plurality of message queues are used for storing a plurality of log files;

the plurality of analysis components are used for analyzing the plurality of log files to obtain a plurality of analyzed log files;

the database is used for storing a plurality of analyzed log files.

Optionally, the plurality of acquisition components are specifically configured to monitor an acquisition state of each log file according to a cache directory;

Optionally, the multiple collecting assemblies are further specifically configured to store the multiple log files in a cache queue according to a collecting time corresponding to each log file, where the collecting time is used to indicate a time corresponding to the collected log file; wherein any one of the log files in the buffer queue is randomly stored to one of the plurality of message queues.

Optionally, the plurality of parsing components are specifically configured to monitor operating states of the plurality of parsing components;

Optionally, the multiple parsing components are specifically configured to store, according to the log identifier corresponding to the multiple parsed log files, each parsed log file into a storage space corresponding to the log identifier in the database.

The invention has the beneficial effects that:

in the embodiment of the invention, the plurality of collecting components are used for collecting the plurality of log files, the collected log files are stored in the plurality of message queues, the plurality of analyzing components can analyze the log files in the plurality of message queues to obtain a plurality of analyzed log files, and finally the analyzed log files are stored in the database. The log file acquisition system has the advantages that the log files are acquired through the acquisition assemblies in the log acquisition system, and the log files are analyzed through the analysis assemblies, so that the condition that the log files cannot be acquired and analyzed when any one acquisition assembly or analysis assembly is abnormal is avoided, the coupling degree in the process of acquiring the log files is reduced, and the reliability and flexibility of acquiring the log files are improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.

Fig. 1 is a schematic structural diagram of a log collection system according to a log collection method provided by the present invention;

fig. 2 is a schematic flowchart of a log collection method according to an embodiment of the present invention;

fig. 3 is a schematic flowchart of a log collection method according to another embodiment of the present invention;

fig. 4 is a schematic diagram of a log collection system according to an embodiment of the present invention;

fig. 5 is a schematic diagram of a log collection device according to an embodiment of the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention.

Fig. 1 is a schematic structural diagram of a log collection system according to a log collection method provided by the present invention; as shown in fig. 1, the log collection system includes: collection component cluster 110, message queue cluster 120, parsing component cluster 130, and database 140.

Wherein collection component cluster 110 may include a plurality of collection components, message queue cluster 120 may include a plurality of message queues, parsing component cluster 130 may include a plurality of parsing components, and database 140 may include a plurality of storage spaces.

In addition, the message queue cluster 120 may further include a main message queue and a standby message queue, where the main message queue is a message queue for storing the log file in each message queue cluster 120, and the standby message queue is a backup of each main message queue, and when any one of the main message queues fails, the log file may be continuously stored through the corresponding standby message queue.

In the process of collecting the log, the collection component cluster 110 may generate a cache directory according to the log identifier corresponding to each log file in the collection directory, and each collection component in the collection component cluster 110 may collect each log file according to the log identifier corresponding to each log file in the cache directory and the collection state corresponding to each log file, update the cache directory according to the collection progress of each log file, and after a certain collection component finishes collecting a certain log file, select any log file whose collection state is the to-be-collected state to continue collecting according to the collection state of each log file in the cache directory.

The collection directory is preset, for example, a user may set the collection directory according to a log file to be collected, so as to obtain a collection directory including log identifiers corresponding to the log files.

Moreover, the corresponding status of each log file may include: the present invention is not limited to this state in the collection, to-be-collected, and the like.

In addition, the log identifier corresponding to each log file is used to identify the log file, for example, the log identifier may represent identification information of the log file, and/or log category information, and/or log source information, that is, information of a device that generates the log file, and of course, the log identifier may also be used to represent other information of the log file, which is not limited in this embodiment of the present invention.

During the process of collecting each log file, the collecting component may randomly store each collected log file in each message queue of the message queue cluster 120. Each parsing component in the parsing component cluster 130 may obtain a different log file from each message queue of the message queue cluster 120, parse the obtained log file to obtain a parsed log file corresponding to each log file, and store the parsed log file in the database 140 in different manners according to the output configuration information of the different parsing components.

Moreover, in the process that the collection component cluster 110 collects each log file according to the cache directory, the cache directory may be updated according to the collection progress of each log file, and if the collection of each corresponding log file in the collection directory is completed, the cache directory may be emptied.

In addition, if the first idle collection component in the collection component cluster 110 is in an abnormal state during the collection process, the collection of the log file may be stopped, the cache directory is updated, the log file currently collected in an interruption manner is marked in the cache directory, and the interruption position indicating the number of the interruption collection lines is marked, so that abnormal position information is generated in the cache directory, and meanwhile, the state of the log file may be updated to a state to be collected, so that other idle collection components in the collection component cluster 110 may continue to collect the log file.

Wherein the idle collection component is used to indicate collection components in the collection component cluster 110 that have not collected log files.

Correspondingly, the collection component cluster 110 may select from the remaining idle collection components, and if the second idle collection component finishes collecting a certain log file and prepares to collect the next log file, the second idle collection component may be selected to search for the log file whose collection is interrupted according to the generated abnormal position information, and continue to collect the log file which is searched according to the number of lines whose collection is interrupted, thereby completing the collection of the log file.

It should be noted that the collection component cluster 110, the message queue cluster 120, the parsing component cluster 130, and the database 140 may be respectively disposed in different devices, or may be integrally disposed in one device, which is not limited in this embodiment of the present invention.

For example, the collection component cluster 110, the message queue cluster 120, the parsing component cluster 130, and the database 140 may be integrally disposed in a server.

Fig. 2 is a schematic flowchart of a log collecting method according to an embodiment of the present invention, which is applied to the log collecting system shown in fig. 1, and as shown in fig. 2, the method includes:

step 201, a plurality of collection components collect a plurality of log files.

In the process of collecting logs, in order to avoid abnormity of the collecting assemblies, a plurality of log files can be collected through the collecting assemblies, and when any one collecting assembly is abnormal, other collecting assemblies can continue to collect the log files.

Therefore, a plurality of log files can be collected through a plurality of collecting components in the log collecting system.

Specifically, after the log collection system starts collecting log files, the log collection system may start the multiple collection assemblies first, and monitor a preset collection directory, thereby generating a cache directory, and then the multiple collection assemblies may collect the log files in a to-be-collected state according to the collection state of each log file in the cache directory, thereby obtaining multiple log files.

Step 202, the plurality of collection components store the plurality of log files to a plurality of message queues.

After the collection of the plurality of log files is completed, the collection component may randomly store the collected log files in a plurality of preset message queues, so that the plurality of log files in the plurality of message queues may be analyzed by the plurality of analysis components in the subsequent step.

Specifically, after the collection component finishes collecting the plurality of log files, the collected log files may be stored in a preset message queue. Since multiple message queues may be included, individual log files may be randomly stored in individual message queues.

For example, if 10 log files are acquired and 5 message queues exist, 2 log files may be stored in each message queue in a round-robin manner, 10 log files may be all stored in any one message queue, and log files with different quantities may be stored in each message queue.

In addition, in practical application, different log files may correspond to different application programs or different devices, and therefore, the log files may be classified according to the log identifiers representing the log sources in the log files, so that the log files of the same type are stored in the same message queue or different message queues.

The log identifier may be used to indicate an application program to which the log file belongs, may also be used to indicate a device to which the log file belongs, and may also be used to indicate other types of classification rules, which is not limited in this embodiment of the present invention.

It should be noted that, for each message queue in the message queue cluster, each message queue may include a main message queue and a standby message queue, and when the main message queue is normal, the log file may be stored through the main message queue. However, when the main message queue is abnormal, the log file needs to be stored through the standby message queue.

And step 203, the plurality of analysis components analyze the log files in the plurality of message queues to obtain a plurality of analyzed log files.

After the plurality of log files are stored in the plurality of message queues, the plurality of analysis components may randomly acquire the stored log files from any one of the message queues and analyze the acquired log files, thereby obtaining a plurality of analyzed log files.

Specifically, if a target analysis component in the plurality of analysis components needs to acquire a next log file from the message queue for analysis after analyzing a certain log file, because the plurality of message queues store the plurality of log files, the target analysis component can acquire the log file which can be read with the least time, so as to quickly analyze the acquired log file, and finally, generate the analyzed data according to a preset format, so as to obtain the analyzed log file.

For example, the target parsing component may obtain, according to a storage address corresponding to a log file read before, a log file stored in a storage address adjacent to the storage address, so as to achieve fast reading of the log file.

Of course, the target parsing component may also obtain the log file from the message queue again according to the message queue where the previous log file is located.

And step 204, the plurality of analysis components store the plurality of analyzed log files in a database.

After the plurality of analysis components analyze the plurality of log files, the plurality of analysis components need to store each analyzed log file in a corresponding storage space in the database according to the output configuration information corresponding to each analysis component and the output mode indicated by each output configuration information.

For example, the parsed log file may be transmitted in a TCP (Transmission Control Protocol), an SNMP (Simple Network Management Protocol), an HTTP (HyperText transfer Protocol), or the like, so as to be transmitted to a database such as Hadoop (a distributed system infrastructure) or an elastic search (a search server), and finally, the parsed log file is stored through each storage space in the database such as Hadoop or elastic search.

In summary, in the log collecting method provided in the embodiments of the present invention, a plurality of collecting components collect a plurality of log files, and store the collected log files in a plurality of message queues, so that a plurality of parsing components can parse the log files in the plurality of message queues to obtain a plurality of parsed log files, and finally store the parsed log files in a database. The log file acquisition system has the advantages that the log files are acquired through the acquisition assemblies in the log acquisition system, and the log files are analyzed through the analysis assemblies, so that the condition that the log files cannot be acquired and analyzed when any one acquisition assembly or analysis assembly is abnormal is avoided, the coupling degree in the process of acquiring the log files is reduced, and the reliability and flexibility of acquiring the log files are improved.

Fig. 3 is a schematic flowchart of a log collecting method according to another embodiment of the present invention, which is applied to the log collecting system shown in fig. 1, and as shown in fig. 3, the method includes:

step 301, monitoring the collection state of each log file according to the cache directory.

The cache directory is generated by monitoring a preset acquisition directory and is used for indicating the acquisition state of each log file. Also, the collection status of the log file may include: the present invention is not limited to this state in the collection, to-be-collected, and the like.

In order to avoid that a plurality of acquisition assemblies simultaneously acquire the same log file, the acquisition state of each log file can be monitored before the log file is acquired, so that the log file in a state to be acquired is acquired.

Therefore, in the process of collecting the log files by the plurality of collecting assemblies in the log collecting system, the collecting state of each log file can be monitored according to the cache directory, and if the collecting state of a certain log file is detected to be the to-be-collected state, any one collecting assembly in the plurality of collecting assemblies can collect the log file in the to-be-collected state in the subsequent steps.

Step 302, for each log file, if the collection state indicates that the log file is in a state to be collected, collecting the log file through any idle collection component in the plurality of collection components.

If a log file is detected to be in a collection state to be collected, any one of the collection assemblies does not collect an idle collection assembly of the log file, and the log file can be collected.

Since the process of collecting the log file in step 302 is similar to that in step 201, it is not described herein again.

It should be noted that, in the process of acquiring a plurality of log files by a plurality of acquisition components, if an abnormal state occurs in the process of acquiring log files by a first idle acquisition component in the plurality of acquisition components, abnormal position information may be generated, and a second idle acquisition component in the plurality of acquisition components may continue to acquire a target log file according to the abnormal position information.

In the process of collecting the log files by the plurality of collecting assemblies, each collecting assembly normally operates to complete the collection of the log files. However, some collection components are also abnormal, and collection of the currently collected log file cannot be continued, so that abnormal position information can be generated, and other idle collection components can continue to collect the log file.

After the collection state of the log file is determined to be the state to be collected, the second idle collection assembly can continue to collect the log file according to the abnormal position information, so that breakpoint continuous collection of the log file can be realized, and collection of the log file is completed.

Specifically, the interruption position in the log file may be determined according to the abnormal position information, that is, when the log file is started to be collected by the second idle collection component, the position of the collected data in the log file is determined, and then the data indicated by the interruption position of the log file may be collected by the second idle collection component, so as to continue collecting the log file.

It should be noted that the acquisition component may be abnormal due to various reasons, wherein the service may be abnormally stopped due to abnormal reasons such as a program itself and a system program failure, for example, an application program loaded in the log acquisition system may have a problem, thereby causing the acquisition component to be abnormal, or the acquisition component may have an abnormality due to a problem of an application program loaded in the acquisition component cluster.

Further, in the process of generating the abnormal position information, the first idle collection component may acquire an interrupt position, and generate the abnormal position information according to the interrupt position. Wherein the interrupt location is used to indicate a location where the first idle collection component interrupts collecting the log file.

Specifically, if it is detected that the first idle collection component is abnormal in the process of collecting the log file by the first idle collection component, the interrupt position, that is, the number of lines that other idle collection components start to continue collecting, may be determined according to the number of lines that the first idle collection component collects the log file, so that abnormal position information may be generated in the cache directory according to the interrupt position.

For example, if the first idle collection component has collected 10 th line of data of the log file and an abnormality occurs while collecting 11 th line of data, the interrupt position may be determined to be 11 th line of data of the log file, and thus abnormality position information indicating that collection needs to be started from the 11 th line of the log file may be generated.

It should be noted that, while the abnormal location information is generated, the state of the log file may be updated to a state of "to be acquired", so that the remaining idle acquisition components may search for the log file from the log file corresponding to the state to be acquired.

In addition, it should be noted that, in practical application, a plurality of log files may be collected simultaneously through the collection component cluster, and the progress of collecting the log files by each collection component is different. Therefore, in the process of collecting the log files, the cache directory can be updated to represent the collection progress corresponding to different log files. The cache directory may be updated every preset time.

Optionally, if it is detected that the collection of any log file in the plurality of log files is completed, the collection state corresponding to the log file collected in the cache directory may be updated to be collected.

For example, the collection component cluster may collect 10 log files simultaneously, each log file has a different size, when the preset duration is 10 seconds, the collection progress of each log file may be updated every 10 seconds, if the 10 th log file is much longer than the 1 st to 9 th log files, the 1 st to 9 th log files may be collected within 10 seconds, and the 10 th log file may be collected within 16 seconds, after the collection progress of each log file is updated within 10 seconds, the collection status of the 1 st to 9 th log files may be displayed as being collected, and the 10 th log file is not collected, the collection progress of the 10 th log file needs to be displayed, for example, the collection completion 60% may be displayed, or collection 600 lines/1000 lines may be displayed.

Further, after the collection of each log file indicated by the collection directory is completed by the collection component cluster, it indicates that the collection of each log file is completed, and the collection progress of each log file does not need to be determined through the cache directory.

Therefore, if the acquisition states corresponding to the plurality of log files in the cache directory are all detected to be acquired, the cache directory is deleted.

Step 303, the plurality of collection components store the plurality of log files to a plurality of message queues.

After the log files are collected by the collecting assemblies, the log files can be stored in the cache queue, and then the collected log files in the cache queue are randomly stored in the message queues in the process of caching the log files.

Optionally, the multiple collection components may store the multiple log files in the cache queue according to the collection time corresponding to each log file.

And the acquisition time is used for indicating the time corresponding to the acquisition completion log file. Moreover, any one log file in the buffer queue can be randomly stored to one of the plurality of message queues.

Specifically, the acquisition component may cache the acquired log files in a preset storage space, and after a certain log file is acquired, the pointer corresponding to the storage address corresponding to the log file may be stored in a preset cache queue, so that each log file is cached in the cache queue according to the acquisition time corresponding to each log file.

Further, in the process of storing the log files through the cache queue, the log files in the storage space indicated by the pointer may be randomly stored in each message queue in combination with the pointer corresponding to each log file in the cache queue.

It should be noted that, in practical applications, the log collection system may include a plurality of message queues, and each message queue may store a different log file. For example, corresponding to step 202, different log files can be divided into a plurality of different types according to the log identifiers of the respective log files, and then one type of log file or a plurality of types of log files can be stored in one message queue.

Of course, each log file in the cache queue may also be randomly stored in a different message queue, which is not limited in the embodiment of the present invention. For example, each log file may be stored in each message queue in a polling manner, or may be stored in a message queue corresponding to a log identifier according to the log identifier corresponding to each log file.

And step 304, analyzing the log files in the message queues by the plurality of analyzing components to obtain a plurality of analyzed log files.

Because the log collection system can comprise a plurality of analysis components, each analysis component can only analyze one log file in the process of analyzing the log file. In order to improve the efficiency of analyzing the log file by the multiple analyzing components, the working state of each analyzing component can be monitored, so that the log file can be analyzed according to the monitoring result.

Optionally, the working states of the multiple parsing components may be monitored, and for each parsing component, if it is detected that the parsing component is in an idle state, a log file in a message queue may be randomly read by the idle parsing component for parsing, so as to obtain a parsed log file.

Additionally, resolving the operational state of the component may include: an idle state to indicate that the parsing component is not parsing the log file, and a parsing state to indicate that the parsing component is parsing the log file.

Specifically, for each analysis component, the log collection system may monitor a working state of the analysis component, and if it is monitored that the analysis component is in an idle state, a log file may be randomly acquired from a plurality of message queues through the analysis component, so as to analyze the acquired log file, and obtain an analyzed log file generated according to a preset format.

Therefore, after the analysis of each log file is completed, a plurality of analyzed log files with uniform formats can be obtained.

And 305, for each analysis component, storing a plurality of analyzed log files in a database by the analysis component according to the output mode indicated by the output configuration information of the analysis component.

Because the output mode of the log file after being analyzed by each analysis component is different, the analysis component can determine the mode of sending the log file after being analyzed to the database according to the output configuration information of the analysis component in the process of storing the log file after being analyzed to the database.

Further, in the process that the plurality of analysis components store the plurality of analyzed log files in the database, the plurality of analysis components may store each analyzed log file in a storage space corresponding to a log identifier in the database according to the log identifier corresponding to the plurality of analyzed log files.

For example, if the plurality of analysis components analyze 10 analyzed log files, the log identifier corresponding to each analyzed log file is shown in table 1, the log identifiers corresponding to the analyzed log files A, C, D, I and J are 1, the log identifiers corresponding to the analyzed log files B, G and H are 2, and the log identifiers corresponding to the analyzed log files E and F are 3, the analyzed log files A, C, D, I and J may be stored in the first storage space, the analyzed log files B, G and H may be stored in the second storage space, and the analyzed log files E and F may be stored in the third storage space.

TABLE 1

Parsed log file	Log identification
		A. C, D, I and J	1
B. G and H	2
		E and F	3

Fig. 4 is a schematic diagram of a log collection system according to an embodiment of the present invention, and as shown in fig. 4, the log collection system specifically includes: a plurality of acquisition components 401, a plurality of message queues 402, a plurality of parsing components 403, and a database 404;

the multiple collection component 401 is used for collecting multiple log files;

the plurality of message queues 402 are used to store a plurality of log files;

the multiple analysis components 403 are configured to analyze multiple log files to obtain multiple analyzed log files;

the database 404 is used to store a plurality of parsed log files.

Optionally, the multiple collection assemblies 401 are specifically configured to monitor a collection state of each log file according to the cache directory;

and for each log file, if the acquisition state indicates that the log file is in a state to be acquired, acquiring the log file through any idle acquisition assembly in the acquisition assemblies.

Optionally, the multiple collecting assemblies 401 are further specifically configured to store the multiple log files in a cache queue according to a collecting time corresponding to each log file, where the collecting time is used to indicate a time corresponding to the completion of collecting the log files; wherein any one of the log files in the buffer queue is randomly stored to one of the plurality of message queues.

Optionally, the multiple analysis components 403 are specifically configured to monitor operating states of the multiple analysis components;

and for each analysis assembly, if the analysis assembly is detected to be in an idle state, randomly reading one log file in one message queue through the idle analysis assembly for analysis to obtain the analyzed log file.

Optionally, the multiple parsing components 403 are specifically configured to store, according to the log identifiers corresponding to multiple parsed log files, each parsed log file into a storage space corresponding to the log identifier in the database.

In summary, in the log collecting system provided in the embodiment of the present invention, the multiple collecting assemblies collect multiple log files, and store the collected log files in the multiple message queues, so that the multiple parsing assemblies can parse the log files in the multiple message queues to obtain multiple parsed log files, and finally store the parsed log files in the database. The log file acquisition system has the advantages that the log files are acquired through the acquisition assemblies in the log acquisition system, and the log files are analyzed through the analysis assemblies, so that the condition that the log files cannot be acquired and analyzed when any one acquisition assembly or analysis assembly is abnormal is avoided, the coupling degree in the process of acquiring the log files is reduced, and the reliability and flexibility of acquiring the log files are improved.

The system is used for executing the method provided by the foregoing embodiment, and the implementation principle and technical effect are similar, which are not described herein again.

These above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above modules is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

Fig. 5 is a schematic diagram of a log collecting apparatus according to an embodiment of the present invention, where the apparatus may be integrated in a terminal device or a chip of the terminal device, and the terminal may be a computing device with a log collecting function.

The device includes: memory 501, processor 502.

The memory 501 is used for storing programs, and the processor 502 calls the programs stored in the memory 501 to execute the above method embodiments. The specific implementation and technical effects are similar, and are not described herein again.

Optionally, the invention also provides a program product, for example a computer-readable storage medium, comprising a program which, when being executed by a processor, is adapted to carry out the above-mentioned method embodiments.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, the division of the units is only one logical division, and other divisions may be realized in practice, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, or in a form of hardware plus a software functional unit.

The integrated unit implemented in the form of a software functional unit may be stored in a computer readable storage medium. The software functional unit is stored in a storage medium and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute some steps of the methods according to the embodiments of the present invention. And the aforementioned storage medium includes: a U disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Claims

1. A log collection method is applied to a log collection system, and the log collection system comprises: the system comprises a plurality of acquisition components, a plurality of analysis components, a plurality of message queues and a database, and the method comprises the following steps:

a plurality of collecting components collect a plurality of log files;

the plurality of analysis components store the plurality of analyzed log files to the database;

the plurality of collection components collect a plurality of log files, including:

for each log file, if the acquisition state indicates that the log file is in a state to be acquired, acquiring the log file through any idle acquisition assembly in a plurality of acquisition assemblies;

any one idle collection assembly in the collection assemblies collects the log file, and the collection assembly comprises:

2. The method of claim 1, wherein the method further comprises:

3. The method of claim 1, wherein the generating anomaly location information comprises:

acquiring an interruption position, wherein the interruption position is used for indicating a position at which the first idle acquisition component interrupts acquisition of the log file;

4. The method of claim 1, wherein the plurality of the collection components storing a plurality of the log files to a plurality of the message queues comprises:

5. The method of claim 1, wherein the parsing the log files in the plurality of message queues by the plurality of parsing components to obtain a plurality of parsed log files comprises:

monitoring the working states of a plurality of analysis components;

6. The method of claim 1, wherein the plurality of parsing components storing a plurality of the parsed log files to the database comprises:

7. A log collection system, comprising: the system comprises a plurality of acquisition components, a plurality of message queues, a plurality of analysis components and a database;

the plurality of message queues are used for storing a plurality of log files;

the database is used for storing a plurality of analyzed log files;

the plurality of acquisition components are specifically used for monitoring the acquisition state of each log file according to a cache directory;

for each log file, if the collection state indicates that the log file is in a state to be collected, collecting the log file through any one idle collection assembly in the collection assemblies, and collecting the log file through any one idle collection assembly in the collection assemblies comprises: if the first idle acquisition assembly in the plurality of acquisition assemblies is in an abnormal state in the process of acquiring the log file, generating abnormal position information, and continuously acquiring the log file according to the abnormal position information through the second idle acquisition assembly in the plurality of acquisition assemblies.

8. The log collection system of claim 7, wherein the collection components are further specifically configured to store the log files in a cache queue according to a collection time corresponding to each log file, where the collection time is used to indicate a time corresponding to the completion of collection of the log files; wherein any one of the log files in the buffer queue is randomly stored to one of the plurality of message queues.

9. The log collection system of claim 7, wherein a plurality of said parsing components are specifically configured to monitor an operating state of a plurality of said parsing components;

10. The log collection system of claim 7, wherein the plurality of parsing components are specifically configured to store each parsed log file into a storage space in the database corresponding to a log identifier according to the log identifier corresponding to the plurality of parsed log files.