CN110011845A - Log collection method and system - Google Patents
Log collection method and system Download PDFInfo
- Publication number
- CN110011845A CN110011845A CN201910255935.7A CN201910255935A CN110011845A CN 110011845 A CN110011845 A CN 110011845A CN 201910255935 A CN201910255935 A CN 201910255935A CN 110011845 A CN110011845 A CN 110011845A
- Authority
- CN
- China
- Prior art keywords
- acquisition
- journal file
- journal
- components
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/12—Network monitoring probes
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides a kind of log collection method and system, is related to field of communication technology.The log collection method includes: that multiple acquisition components acquire multiple journal files, multiple acquisition components store multiple journal files to multiple message queues, multiple resolution components parse the journal file in multiple message queues, journal file after obtaining multiple parsings, multiple resolution components store the journal file after multiple parsings to the database.Multiple journal files are acquired by multiple acquisition components in Log Collect System, and journal file is parsed by multiple resolution components, it avoids any one acquisition component or the case where can not being acquired analysis to journal file when exception occurs in resolution component, the degree of coupling during acquisition journal file is reduced, the reliability and flexibility of acquisition journal file are improved.
Description
Technical field
The present invention relates to fields of communication technology, in particular to a kind of log collection method and system.
Background technique
With the continuous development of internet and big data, the log that the network equipment generates is also more and more, and user can be with base
Big data analysis is carried out in log, obtains valuable data in log.
In the related technology, it can be acquired, and led to by the journal file that acquisition component generates each network equipment
It crosses resolution component to parse the journal file collected, finally stores the data that parsing obtains to pre-set number
According in storage system.
But the process for acquiring log is linear correlation, the degree of coupling is higher, when acquisition component, resolution component or data
When storage system occurs abnormal, leads to not normal acquisition and store log.
Summary of the invention
It is an object of the present invention in view of the deficiency of the prior art, a kind of log collection method and device are provided,
It is higher with the degree of coupling for solving acquisition log, when acquisition component, resolution component or data-storage system occur abnormal, Wu Fazheng
The problem of often acquiring and store log.
To achieve the above object, technical solution used in the embodiment of the present invention is as follows:
In a first aspect, being applied to Log Collect System, the day the embodiment of the invention provides a kind of log collection method
Will acquisition system includes: multiple acquisition components, multiple resolution components, multiple message queues and database, which comprises
Multiple acquisition components acquire multiple journal files；
Multiple acquisition components store multiple journal files to multiple message queues；
Multiple resolution components parse the journal file in multiple message queues, obtain multiple solutions
Journal file after analysis；
Multiple resolution components store the journal file after multiple parsings to the database.
Optionally, the multiple acquisition component acquires multiple journal files, comprising:
The acquisition state of each journal file is monitored according to CACHE DIRECTORY；
Each journal file is led to if the acquisition state indicates that the journal file is in state to be collected
Any one the idle acquisition component crossed in multiple acquisition components is acquired the journal file.
Optionally, the method also includes:
Any one journal file acquisition in multiple journal files is finished if detecting, by the caching mesh
Acquired in record acquisition state corresponding to the journal file finished be updated to acquisition finish；
If detecting, the corresponding acquisition state of the multiple journal files is that acquisition finishes in the CACHE DIRECTORY, is deleted
Except the CACHE DIRECTORY.
Optionally, the idle acquisition component of any one in the multiple acquisition component carries out the journal file
Acquisition, comprising:
If the first idle acquisition component in multiple acquisition components occurs during acquiring the journal file
Abnormality generates abnormal position information；
The second idle acquisition component in multiple acquisition components, continues according to the abnormal position information to the day
Will file is acquired.
Optionally, the generation abnormal position information, comprising:
Interruption position is obtained, the interruption position is used to indicate log described in the described first idle acquisition component interrupt acquisition
The position of file；
The abnormal position information is generated according to the interruption position.
Optionally, the multiple acquisition component stores multiple journal files to multiple message queues,
Include:
Multiple acquisition components are according to each journal file corresponding acquisition moment, by multiple journal files
It is stored in buffer queue, at the time of the acquisition moment is used to indicate corresponding to acquisition completion journal file；Wherein, described slow
Deposit a message queue of any one of journal file by random storage into multiple message queues in queue.
Optionally, the multiple resolution component solves the journal file in multiple message queues
Analysis, the journal file after obtaining multiple parsings, comprising:
Monitor the working condition of multiple resolution components；
For each resolution component, if detecting, the resolution component is in idle condition, and passes through the idle solution
Analysis component reads a journal file in the message queue at random and is parsed, the day after obtaining the parsing
Will file.
Optionally, the multiple resolution component stores the journal file after multiple parsings to the data
Library, comprising:
Multiple resolution components log according to corresponding to the journal file after multiple parsings mark, by each institute
Journal file after stating parsing stores into the database memory space corresponding with log mark.
Second aspect, the embodiment of the invention also provides a kind of Log Collect System, the Log Collect System includes: more
A acquisition component, multiple message queues, multiple resolution components and database；
Multiple acquisition components are for acquiring multiple journal files；
Multiple message queues are for storing multiple journal files；
Log text of multiple resolution components for being parsed to multiple journal files, after obtaining multiple parsings
Part；
The database is for storing the journal file after multiple parsings.
Optionally, multiple acquisition components, specifically for monitoring adopting for each journal file according to CACHE DIRECTORY
Collection state；
Each journal file is led to if the acquisition state indicates that the journal file is in state to be collected
Any one the idle acquisition component crossed in multiple acquisition components is acquired the journal file.
Optionally, multiple acquisition components, also particularly useful for according to each journal file corresponding acquisition moment,
Multiple journal files are stored in buffer queue, the acquisition moment is used to indicate acquisition and completes corresponding to journal file
At the time of；Wherein, any one of journal file in the buffer queue is by random storage to multiple message queues
In a message queue.
Optionally, multiple resolution components, specifically for monitoring the working condition of multiple resolution components；
For each resolution component, if detecting, the resolution component is in idle condition, and passes through the idle solution
Analysis component reads a journal file in the message queue at random and is parsed, the day after obtaining the parsing
Will file.
Optionally, multiple resolution components, specifically for according to corresponding to the journal file after multiple parsings
Log mark, stores into the database deposit corresponding with log mark for the journal file after each parsing
Store up space.
The beneficial effects of the present invention are:
The log that the embodiment of the present invention is acquired to multiple journal files by multiple acquisition components, and will collected
File is stored in multiple message queues, then multiple resolution components can solve the journal file in multiple message queues
Analysis, the journal file after obtaining multiple parsings, finally in the database by the journal file storage after parsing.Pass through log collection
Multiple acquisition components in system are acquired multiple journal files, and are solved by multiple resolution components to journal file
Analysis, avoids any one acquisition component or resolution component occurs that journal file can not be acquired the feelings of analysis when exception
Condition reduces the degree of coupling during acquisition journal file, improves the reliability and flexibility of acquisition journal file.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below will be to needed in the embodiment attached
Figure is briefly described, it should be understood that the following drawings illustrates only certain embodiments of the present invention, therefore is not construed as pair
The restriction of range for those of ordinary skill in the art without creative efforts, can also be according to this
A little attached drawings obtain other relevant attached drawings.
Fig. 1 is the structural schematic diagram of Log Collect System involved in a kind of log collection method provided by the invention；
Fig. 2 is the flow diagram for the log collection method that one embodiment of the invention provides；
Fig. 3 be another embodiment of the present invention provides log collection method flow diagram；
Fig. 4 is the schematic diagram for the Log Collect System that one embodiment of the invention provides；
Fig. 5 is the schematic diagram for the log acquisition apparatus that one embodiment of the invention provides.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present invention
In attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment is
A part of the embodiment of the present invention, instead of all the embodiments.
Fig. 1 is the structural schematic diagram of Log Collect System involved in a kind of log collection method provided by the invention；Such as
Shown in Fig. 1, which includes: acquisition component cluster 110, message queue cluster 120,130 and of resolution component cluster
Database 140.
Wherein, acquisition component cluster 110 may include multiple acquisition components, and message queue cluster 120 may include multiple
Message queue, resolution component cluster 130 may include multiple resolution components, and database 140 may include multiple memory spaces.
In addition, message queue cluster 120 can also include main message queue and standby message queue, main message queue is message
It is each for storing the message queue of journal file in queue cluster 120, and standby message queue is then each main message queue
Backup can continue to store journal file by corresponding standby message queue when any one main message queue breaks down.
During acquiring log, acquisition component cluster 110 can be corresponding according to each journal file in acquisition catalogue
Log mark generate CACHE DIRECTORY, moreover, each acquisition component in acquisition component cluster 110 can be according in CACHE DIRECTORY
The corresponding log mark of each journal file and the corresponding acquisition state of each journal file, adopt each journal file
Collection, and according to the acquisition progress of each journal file, CACHE DIRECTORY is updated, if the acquisition of some acquisition component finishes some
After journal file, can according to the acquisition state of journal file each in CACHE DIRECTORY, choose any one acquisition state be to
The journal file of acquisition state continues to acquire.
Wherein, which is pre-set, for example, user can according to need the journal file of acquisition, to adopting
Collection catalogue is configured, so that obtaining includes acquisition catalogue that the corresponding log of each journal file identifies.
Moreover, the states such as the corresponding state of each journal file may include: in acquisition, to be collected and acquisition finishes, this
Inventive embodiments do not limit this.
In addition, the corresponding log mark of each journal file is for being identified journal file, for example, the log identifies
The identification information and/or log information and/or log source information that can indicate journal file that is to say and generate log text
The information of the equipment of part, certain log mark can be also used for indicating the other information of journal file, the embodiment of the present invention pair
This is without limitation.
During being acquired to each journal file, each journal file that acquisition component can will collect
Random storage is in each message queue of message queue cluster 120.And each resolution component in resolution component cluster 130 is then
Can obtain different journal files from each message queue of message queue cluster 120, and to the journal file of acquisition into
Row parsing, to obtain the journal file after the corresponding parsing of each journal file, the output further according to different resolution components is matched
Journal file after parsing, is stored in database 140 by confidence breath in different ways.
It, can be with moreover, during acquisition component cluster 110 is acquired each journal file according to CACHE DIRECTORY
According to the acquisition progress of each journal file, CACHE DIRECTORY is updated, if to each log text corresponding in acquisition catalogue
Part is acquired and is finished, then can empty the cache catalogue.
In addition, if abnormality occurs in collection process in the first idle acquisition component in acquisition component cluster 110,
It can stop acquiring journal file, and CACHE DIRECTORY is updated, to the log text of Current interrupt acquisition in CACHE DIRECTORY
Part is marked, and is labeled to the interruption position of expression interrupt acquisition line number, to generate exception bits in CACHE DIRECTORY
Confidence breath, at the same time it can also which the state of the journal file to be updated to state to be collected, so as in acquisition component cluster 110
Other idle acquisition components the journal file can be continued to acquire.
Wherein, which is used to indicate the acquisition group that journal file is not acquired in acquisition component cluster 110
Part.
Corresponding, acquisition component cluster 110 can be chosen from remaining idle acquisition component, if second is idle
Acquisition component has acquired some journal file, and the next journal file of preparation for acquiring can then choose the second idle acquisition component
According to the abnormal position information of generation, lookup obtains the journal file of interrupt acquisition, and according to the line number of interrupt acquisition, continues pair
It searches obtained journal file to be acquired, to complete the acquisition to the journal file.
It should be noted that above-mentioned acquisition component cluster 110, message queue cluster 120,130 sum number of resolution component cluster
It can be separately positioned on according to library 140 in different equipment, also can integrate and be arranged in an equipment, the embodiment of the present invention is to this
Without limitation.
For example, acquisition component cluster 110, message queue cluster 120, resolution component cluster 130 and database 140 can collect
In the server at setting.
Fig. 2 is the flow diagram for the log collection method that one embodiment of the invention provides, applied to day as shown in Figure 1
Will acquisition system, as shown in Fig. 2, this method comprises:
Step 201, multiple acquisition components acquire multiple journal files.
It, can be by multiple acquisition components to more in order to avoid exception occurs in acquisition component during acquiring log
A journal file is acquired, and when any one acquisition component occurs abnormal, other acquisition components can continue to log text
Part is acquired.
Therefore, multiple journal files can be acquired by multiple acquisition components in Log Collect System.
Specifically, Log Collect System can first start multiple acquisition components, and to pre- after starting to acquire journal file
The acquisition catalogue being first arranged is monitored, to generate CACHE DIRECTORY, then multiple acquisition components can be according to each in CACHE DIRECTORY
The acquisition state of a journal file is acquired each journal file in state to be collected, to obtain multiple logs
File.
Step 202, multiple acquisition components store multiple journal files to multiple message queues.
After being acquired to multiple journal files, then acquisition component can will collect journal file random storage
In pre-set multiple message queues, so as to can be by multiple resolution components in multiple message queues in subsequent step
Multiple journal files parsed.
Specifically, acquisition component is after acquiring multiple journal files, then the multiple logs that can will collect
File is stored in pre-set message queue.It, then can be by each journal file due to may include multiple message queues
Random storage is in each message queue.
For example, if collect 10 journal files, and there are 5 message queues, then can by the way of repeating query
2 journal files are stored in each message queue, 10 journal files can also be stored entirely in any one message queue
In, the different journal file of storage quantity, the embodiment of the present disclosure can also be stored to message queue day in each message queue
The mode of will file is without limitation.
In addition, in practical applications, different journal files can correspond to different application programs or correspond to different set
It is standby, therefore, it can identify according to the log of Log Source is indicated in journal file, classify to each journal file, thus
The journal file of same type is stored in same message queue or different messages queue.
Wherein, log mark can serve to indicate that application program belonging to journal file, can be used for instruction log text
Equipment belonging to part may be used to indicate that other kinds of classifying rules, and it is not limited in the embodiment of the present invention.
It should be noted that each message queue may include master for each message queue in message queue cluster
Message queue and standby message queue can store journal file by main message queue when main message queue is normal.
But when main message queue appearance exception, then need to store journal file by standby message queue.
Step 203, multiple resolution components parse the journal file in multiple message queues, after obtaining multiple parsings
Journal file.
It stores by multiple journal files to multiple message queues, multiple resolution components can be from any one message team
The journal file of storage is obtained in column at random, and the journal file of acquisition is parsed, to obtain the day after multiple parsings
Will file.
Specifically, if needing after the target resolution component in multiple resolution components parsed some journal file from message
Next journal file is obtained in queue to be parsed, due to being stored with multiple journal files in multiple message queues, then target
The available journal file that the minimum time can be spent to read of resolution component, to quickly be solved to the journal file of acquisition
Analysis finally generates the data after parsing according to pre-set format, thus the journal file after being parsed.
For example, target resolution component can be obtained and be deposited with this according to the corresponding storage address of the journal file read before
The journal file stored in the adjacent storage address in storage address quickly reads journal file to realize.
Certainly, target resolution component can also be according to the message queue where a upper journal file, again from the message team
Journal file is obtained in column, the embodiment of the present invention obtains the mode of journal file without limitation to resolution component.
Step 204, multiple resolution components store the journal file after multiple parsings to database.
After multiple resolution components parse multiple journal files, then need corresponding defeated according to each resolution component
Journal file after each parsing is stored in by configuration information out by the way of output indicated by each output configuration information
Corresponding memory space in database.
For example, TCP (Transmission Control Protocol transmission control protocol), SNMP can be used
(Simple Network Management Protocol, Simple Network Management Protocol) or HTTP (HyperText
Transport Protocol, hypertext transfer protocol) etc. modes the journal file after parsing is transmitted, to be transmitted to
The databases such as Hadoop (a kind of distributed system infrastructure) or ElasticSearch (a kind of search server), finally lead to
The each memory space crossed in the databases such as Hadoop or ElasticSearch stores the journal file after parsing.
In conclusion log collection method provided in an embodiment of the present invention, by multiple acquisition components to multiple log texts
Part is acquired, and the journal file collected is stored in multiple message queues, then multiple resolution components can be to more
Journal file in a message queue is parsed, the journal file after obtaining multiple parsings, finally by the log text after parsing
Part stores in the database.Multiple journal files are acquired by multiple acquisition components in Log Collect System, and are led to
Multiple resolution components are crossed to parse journal file, avoid when any one acquisition component or resolution component occur abnormal without
The case where method is acquired analysis to journal file reduces the degree of coupling during acquisition journal file, improves acquisition day
The reliability and flexibility of will file.
Fig. 3 be another embodiment of the present invention provides log collection method flow diagram, applied to as shown in Figure 1
Log Collect System, as shown in figure 3, this method comprises:
Step 301, the acquisition state that each journal file is monitored according to CACHE DIRECTORY.
Wherein, CACHE DIRECTORY is used to indicate each by generating after monitoring pre-set acquisition catalogue
The acquisition state of journal file.Moreover, the acquisition state of journal file may include: in acquisition, to be collected and acquisition finishes
State, it is not limited in the embodiment of the present invention.
In order to avoid multiple acquisition components are simultaneously acquired the same journal file, before acquiring journal file,
The acquisition state of each journal file can be monitored, to be acquired to the journal file in state to be collected.
It therefore, can first root during multiple acquisition components in Log Collect System are acquired journal file
Be monitored according to acquisition state of the CACHE DIRECTORY to each journal file, if detect some journal file acquisition state be to
Acquisition state, then in the next steps, any one acquisition component in multiple acquisition components can be in state to be collected
Journal file be acquired.
Step 302, for each journal file, if acquisition state instruction journal file is in state to be collected, by more
Any one idle acquisition component in a acquisition component is acquired journal file.
If detecting, some journal file is in acquisition state to be collected, any one in multiple acquisition components is simultaneously
The idle acquisition component of journal file is not acquired, which can be acquired.
Since the process that this step 302 acquires journal file is similar with the process of step 201, details are not described herein.
It should be noted that during multiple acquisition components acquire multiple journal files, if in multiple acquisition components
The first idle acquisition component there is abnormality during acquiring journal file, abnormal position information can be generated, and
The second idle acquisition component in multiple acquisition components, then can be continued according to the abnormal position information to target journaling file into
Row acquisition.
During multiple acquisition components are acquired journal file, each acquisition component would generally be operated normally,
Complete the acquisition to journal file.But part acquisition component also will appear exception, can not continue to the log text currently acquired
Part is acquired, so as to generate abnormal position information, so that other free time acquisition components can continue to acquire log text
Part.
After the acquisition state for determining journal file is state to be collected, the second idle acquisition component can be according to exception bits
Confidence breath, continues to be acquired journal file, so as to realize it is continuous to the breakpoint of journal file adopt, complete to journal file
Acquisition.
Specifically, it can determine the interruption position in journal file first according to abnormal position information, that is to say, pass through
When two idle acquisition components start to be acquired journal file, position of the collected data in journal file is determined, then
Can be by the second idle acquisition component, data indicated by the interruption position to journal file are acquired, to continue to adopt
Collect journal file.
It should be noted that may cause acquisition component abnormal due to many reasons, wherein may be due to program itself
The reason of, the improper reason such as system program failure cause service to abend, for example, what is loaded in Log Collect System answers
With program it is possible that problem, to cause acquisition component abnormal, it is also possible to since what acquisition component cluster was loaded answers
It is gone wrong with program, causes acquisition component to occur abnormal, the embodiment of the present invention does not do the reason for causing acquisition component exception
It limits.
Further, during generating abnormal position information, the first available interruption position of idle acquisition component,
And abnormal position information is generated according to the interruption position.Wherein, which is used to indicate the first idle acquisition component and interrupts
Acquire the position of journal file.
Specifically, if during the first idle acquisition component acquisition journal file, the first idle acquisition group is detected
There is abnormal, then can to acquire according to the first idle acquisition component to journal file line number in part, determines interruption position, that is to say
Other idle acquisition components start to continue the line number of acquisition, abnormal so as to be generated in CACHE DIRECTORY according to the interruption position
Location information.
For example, acquiring the 11st row data if the first idle acquisition component has acquired 10 row data of journal file
When, occurs abnormal, then can determine that interruption position is the 11st row data of journal file, need so as to generate instruction from day
11st row of will file starts the abnormal position information of acquisition.
It should be noted that the state of journal file can also be updated to while generating abnormal position information " to
The state of acquisition " obtains the day so that remaining idle acquisition component can be searched from the corresponding journal file of state to be collected
Will file.
It, in practical applications, can be by acquisition component cluster, while to multiple logs text in addition, it is necessary to explanation
Part is acquired, and the progress of each acquisition component acquisition journal file is different.Therefore, in the process of acquisition journal file
In, CACHE DIRECTORY can be updated, to indicate the corresponding acquisition progress of different journal files.Wherein it is possible to every default
Duration is updated CACHE DIRECTORY.
Optionally, any one journal file acquisition in multiple journal files is finished if detecting, can will be delayed
Deposit acquisition state corresponding to the journal file for acquiring and finishing in catalogue be updated to acquisition finish.
For example, acquisition component cluster can acquire 10 journal files simultaneously, each journal file it is of different sizes, when pre-
If when a length of 10 seconds when, then the acquisition progress of each journal file can be updated every 10 seconds, if the 10th log text
Part is much larger than the 1st to the 9th journal file, and the 1st to the 9th journal file can acquire in 10 seconds to be finished, and the 10th
A journal file, which then needs to acquire for 16 seconds, to be finished, then is updated by 10 seconds to the acquisition progress of each journal file
Afterwards, it can show that the acquisition state of the 1st to the 9th journal file finishes for acquisition, and the 10th journal file does not acquire
It finishes, then needs to show the acquisition progress of the 10th journal file, for example, can show that acquisition completes 60%, or display is adopted
Collect totally 1000 row of 600 rows/.
Further, if after acquisition component cluster acquires each journal file indicated by acquisition catalogue,
Illustrate to have been completed the acquisition to each journal file, then no longer needs to the acquisition for determining each journal file by CACHE DIRECTORY
Progress.
Therefore, if detecting, the corresponding acquisition state of multiple journal files is that acquisition finishes in CACHE DIRECTORY, is deleted
The CACHE DIRECTORY.
Step 303, multiple acquisition components store multiple journal files to multiple message queues.
Each journal file first can be stored in buffer queue after starting acquisition to journal file by multiple acquisition components
In, then during caching each journal file, the journal file random storage finished that acquires each in buffer queue is existed
In each message queue.
Optionally, multiple acquisition components can be according to each journal file corresponding acquisition moment, by multiple journal files
It is stored in buffer queue.
Wherein, at the time of which is used to indicate corresponding to acquisition completion journal file.Moreover, in buffer queue
Any one journal file can be by a message queue of the random storage into multiple message queues.
Specifically, the journal file collected can be first buffered in pre-set memory space by acquisition component,
If pointer corresponding to the corresponding storage address of the journal file can be stored in after some journal file acquires
In pre-set buffer queue, so that each journal file is buffered according to each journal file corresponding acquisition moment
In buffer queue.
It further, can be with each day in binding cache queue during storing journal file by buffer queue
The corresponding pointer of will file, by the journal file random storage in memory space indicated by pointer in each message queue.
It should be noted that in practical applications, Log Collect System may include multiple message queues, and each message
Queue can store different journal files.For example, it is corresponding with step 202, it can be according to the log of each journal file
Different journal files, is divided into multiple and different types by mark, then can be by the journal file of type or multiple classes
The journal file of type is stored in a message queue.
Certainly, each journal file in buffer queue, can also be with random storage in different message queues, the present invention
Embodiment does not limit this.For example, each journal file can be stored in each message queue with poll, it can also be according to each
The corresponding log mark of a journal file, is stored in message queue corresponding with log mark.
Step 304, multiple resolution components parse the journal file in multiple message queues, after obtaining multiple parsings
Journal file.
Due to may include multiple resolution components in Log Collect System, and each resolution component is in parsing journal file
In the process, a journal file can only be parsed.Journal file is parsed in order to improve multiple resolution components
Efficiency can be monitored the working condition of each resolution component, to be parsed according to monitoring result to journal file.
Optionally, the working condition that multiple resolution components can be monitored, for each resolution component, if detecting the parsing
Component is in idle condition, and the journal file that can be read at random by idle resolution component in a message queue carries out
Parsing, the journal file after being parsed.
In addition, the working condition of resolution component may include: idle state and analysis state, idle state is used to indicate solution
Analysis component does not parse journal file, and analysis state is then used to indicate resolution component and solves to journal file
Analysis.
Specifically, for each resolution component, Log Collect System can monitor the working condition of resolution component, if monitoring
It is in idle condition to the resolution component, then can obtain a log at random from multiple message queues by the resolution component
File, so that the journal file to acquisition parses, the log text after obtaining the parsing generated according to pre-set format
Part.
Therefore, the log text after being parsed to each journal file, after the parsing of available multiple unified formats
Part.
Step 305, for each resolution component, resolution component output side according to indicated by the output configuration information of itself
Formula stores the journal file after multiple parsings to database.
Since the mode of the journal file after the output parsing of each resolution component is different, resolution component will parse
During journal file afterwards is stored to database, it can be determined according to the output configuration information of resolution component itself to number
The mode of the journal file after parsing is sent according to library.
Further, more during multiple resolution components store the journal file after multiple parsings to database
A resolution component can the log according to corresponding to the journal file after multiple parsings mark, by the journal file after each parsing
Memory space corresponding with log mark is stored into database.
For example, if multiple resolution components parse to obtain the journal file after 10 parsings, the journal file after each parsing
As shown in table 1, the corresponding log of journal file A, C, D, I and J after parsing is identified as 1, after parsing to corresponding log mark
The corresponding log of journal file B, G and H be identified as 2, the corresponding log of journal file E and F after parsing is identified as 3, then may be used
Journal file A, C, D, I and J after parsing are stored in the first memory space, and journal file B, G and H after parsing are deposited
Storage is stored in third memory space in the second memory space, then by the journal file E and F after parsing.
Table 1
| Journal file after parsing | Log mark |
| A, C, D, I and J | 1 |
| B, G and H | 2 |
| E and F | 3 |
In conclusion log collection method provided in an embodiment of the present invention, by multiple acquisition components to multiple log texts
Part is acquired, and the journal file collected is stored in multiple message queues, then multiple resolution components can be to more
Journal file in a message queue is parsed, the journal file after obtaining multiple parsings, finally by the log text after parsing
Part stores in the database.Multiple journal files are acquired by multiple acquisition components in Log Collect System, and are led to
Multiple resolution components are crossed to parse journal file, avoid when any one acquisition component or resolution component occur abnormal without
The case where method is acquired analysis to journal file reduces the degree of coupling during acquisition journal file, improves acquisition day
The reliability and flexibility of will file.
Fig. 4 is the schematic diagram for the Log Collect System that one embodiment of the invention provides, as shown in figure 4, the log collection system
System specifically includes: multiple acquisition components 401, multiple message queues 402, multiple resolution components 403 and database 404；
Multiple acquisition components 401 are for acquiring multiple journal files；
Multiple message queues 402 are for storing multiple journal files；
Multiple resolution components 403 are for parsing multiple journal files, the journal file after obtaining multiple parsings；
Database 404 is for storing the journal file after multiple parsings.
Optionally, multiple acquisition components 401, specifically for monitoring the acquisition shape of each journal file according to CACHE DIRECTORY
State；
For each journal file, if the acquisition state indicates that the journal file is in state to be collected, by multiple
Any one idle acquisition component in the acquisition component is acquired the journal file.
Optionally, multiple acquisition components 401 are incited somebody to action also particularly useful for according to each journal file corresponding acquisition moment
Multiple journal files are stored in buffer queue, when which is used to indicate corresponding to acquisition completion journal file
It carves；Wherein, any one journal file in the buffer queue is disappeared by one into multiple message queues of random storage
Cease queue.
Optionally, multiple resolution components 403, specifically for monitoring the working condition of multiple resolution components；
For each resolution component, if detecting, the resolution component is in idle condition, by the free time resolution component with
Machine-readable journal file taken in the message queue is parsed, the journal file after obtaining the parsing.
Optionally, multiple resolution components 403, specifically for the log according to corresponding to the journal file after multiple parsings
Mark, stores into the database memory space corresponding with log mark for the journal file after each parsing.
In conclusion Log Collect System provided in an embodiment of the present invention, by multiple acquisition components to multiple log texts
Part is acquired, and the journal file collected is stored in multiple message queues, then multiple resolution components can be to more
Journal file in a message queue is parsed, the journal file after obtaining multiple parsings, finally by the log text after parsing
Part stores in the database.Multiple journal files are acquired by multiple acquisition components in Log Collect System, and are led to
Multiple resolution components are crossed to parse journal file, avoid when any one acquisition component or resolution component occur abnormal without
The case where method is acquired analysis to journal file reduces the degree of coupling during acquisition journal file, improves acquisition day
The reliability and flexibility of will file.
The method that above system is used to execute previous embodiment offer, it is similar that the realization principle and technical effect are similar, herein not
It repeats again.
The above module can be arranged to implement one or more integrated circuits of above method, such as: one
Or multiple specific integrated circuits (Application Specific Integrated Circuit, abbreviation ASIC), or, one
Or multi-microprocessor (digital singnal processor, abbreviation DSP), or, one or more field programmable gate
Array (Field Programmable Gate Array, abbreviation FPGA) etc..For another example, when some above module passes through processing elements
When the form of part scheduler program code is realized, which can be general processor, such as central processing unit (Central
Processing Unit, abbreviation CPU) or it is other can be with the processor of caller code.For another example, these modules can integrate
Together, it is realized in the form of system on chip (system-on-a-chip, abbreviation SOC).
Fig. 5 is the schematic diagram for the log acquisition apparatus that one embodiment of the invention provides, which can integrate sets in terminal
Standby or terminal device chip, the terminal can be the calculating equipment for having log collection function.
The device includes: memory 501, processor 502.
Memory 501 is for storing program, the program that processor 502 calls memory 501 to store, to execute the above method
Embodiment.Specific implementation is similar with technical effect, and which is not described herein again.
Optionally, the present invention also provides a kind of program product, such as computer readable storage medium, including program, the journeys
Sequence is when being executed by processor for executing above method embodiment.
In several embodiments provided by the present invention, it should be understood that disclosed device and method can pass through it
Its mode is realized.For example, the apparatus embodiments described above are merely exemplary, for example, the division of the unit, only
Only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components can be tied
Another system is closed or is desirably integrated into, or some features can be ignored or not executed.Another point, it is shown or discussed
Mutual coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING or logical of device or unit
Letter connection can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of hardware adds SFU software functional unit.
The above-mentioned integrated unit being realized in the form of SFU software functional unit can store and computer-readable deposit at one
In storage media.Above-mentioned SFU software functional unit is stored in a storage medium, including some instructions are used so that a computer
Equipment (can be personal computer, server or the network equipment etc.) or processor (English: processor) execute this hair
The part steps of bright each embodiment the method.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory
(English: Read-Only Memory, abbreviation: ROM), random access memory (English: Random Access Memory, letter
Claim: RAM), the various media that can store program code such as magnetic or disk.
Claims (13)
1. a kind of log collection method, which is characterized in that be applied to Log Collect System, the Log Collect System includes: more
A acquisition component, multiple resolution components, multiple message queues and database, which comprises
Multiple acquisition components acquire multiple journal files；
Multiple acquisition components store multiple journal files to multiple message queues；
Multiple resolution components parse the journal file in multiple message queues, after obtaining multiple parsings
Journal file；
Multiple resolution components store the journal file after multiple parsings to the database.
2. the method as described in claim 1, which is characterized in that the multiple acquisition component acquires multiple journal files,
Include:
The acquisition state of each journal file is monitored according to CACHE DIRECTORY；
For each journal file, if the acquisition state indicates that the journal file is in state to be collected, by more
Any one idle acquisition component in a acquisition component is acquired the journal file.
3. method according to claim 2, which is characterized in that the method also includes:
Any one journal file acquisition in multiple journal files is finished if detecting, it will be in the CACHE DIRECTORY
Acquisition state corresponding to the journal file that acquisition finishes is updated to acquisition and finishes；
If detecting, the corresponding acquisition state of the multiple journal files is that acquisition finishes in the CACHE DIRECTORY, deletes institute
State CACHE DIRECTORY.
4. method according to claim 2, which is characterized in that any one free time in the multiple acquisition component adopts
Collection component is acquired the journal file, comprising:
If the first idle acquisition component in multiple acquisition components occurs abnormal during acquiring the journal file
State generates abnormal position information；
The second idle acquisition component in multiple acquisition components, continues according to the abnormal position information to the log text
Part is acquired.
5. method as claimed in claim 4, which is characterized in that the generation abnormal position information, comprising:
Interruption position is obtained, the interruption position is used to indicate journal file described in the described first idle acquisition component interrupt acquisition
Position；
The abnormal position information is generated according to the interruption position.
6. the method as described in claim 1, which is characterized in that the multiple acquisition component is by multiple journal files
It stores to multiple message queues, comprising:
Multiple acquisition components save multiple journal files according to each journal file corresponding acquisition moment
In buffer queue, at the time of the acquisition moment is used to indicate corresponding to acquisition completion journal file；Wherein, the caching team
A message queue of any one of journal file by random storage into multiple message queues in column.
7. the method as described in claim 1, which is characterized in that the multiple resolution component is to multiple message queues
In the journal file parsed, the journal file after obtaining multiple parsings, comprising:
Monitor the working condition of multiple resolution components；
For each resolution component, if detecting, the resolution component is in idle condition, and passes through the idle parsing group
Part reads a journal file in the message queue at random and is parsed, the log text after obtaining the parsing
Part.
8. the method as described in claim 1, which is characterized in that the multiple resolution component will be after multiple parsings
Journal file is stored to the database, comprising:
Multiple resolution components log according to corresponding to the journal file after multiple parsings mark, by each solution
Journal file after analysis stores into the database memory space corresponding with log mark.
9. a kind of Log Collect System, which is characterized in that the Log Collect System includes: multiple acquisition components, multiple message
Queue, multiple resolution components and database；
Multiple acquisition components are for acquiring multiple journal files；
Multiple message queues are for storing multiple journal files；
Multiple resolution components are for parsing multiple journal files, the journal file after obtaining multiple parsings；
The database is for storing the journal file after multiple parsings.
10. Log Collect System as claimed in claim 9, which is characterized in that multiple acquisition components are specifically used for basis
CACHE DIRECTORY monitors the acquisition state of each journal file；
For each journal file, if the acquisition state indicates that the journal file is in state to be collected, by more
Any one idle acquisition component in a acquisition component is acquired the journal file.
11. Log Collect System as claimed in claim 9, which is characterized in that multiple acquisition components, also particularly useful for pressing
According to each journal file corresponding acquisition moment, multiple journal files are stored in buffer queue, the acquisition
At the time of moment is used to indicate corresponding to acquisition completion journal file；Wherein, any one of day in the buffer queue
A message queue of the will file by random storage into multiple message queues.
12. Log Collect System as claimed in claim 9, which is characterized in that multiple resolution components are specifically used for monitoring
The working condition of multiple resolution components；
For each resolution component, if detecting, the resolution component is in idle condition, and passes through the idle parsing group
Part reads a journal file in the message queue at random and is parsed, the log text after obtaining the parsing
Part.
13. Log Collect System as claimed in claim 9, which is characterized in that multiple resolution components are specifically used for basis
The mark of log corresponding to journal file after multiple parsings, the journal file after each parsing is stored to described
Memory space corresponding with log mark in database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910255935.7A CN110011845B (en) | 2019-03-29 | 2019-03-29 | Log collection method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910255935.7A CN110011845B (en) | 2019-03-29 | 2019-03-29 | Log collection method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110011845A true CN110011845A (en) | 2019-07-12 |
| CN110011845B CN110011845B (en) | 2022-05-10 |
Family
ID=67169222
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910255935.7A Active CN110011845B (en) | 2019-03-29 | 2019-03-29 | Log collection method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110011845B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113111137A (en) * | 2021-04-30 | 2021-07-13 | 深圳壹账通智能科技有限公司 | Distributed system log real-time display method, device, equipment and storage medium |
| CN114817187A (en) * | 2021-01-19 | 2022-07-29 | 成都鼎桥通信技术有限公司 | Log processing method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102158355A (en) * | 2011-03-11 | 2011-08-17 | 广州蓝科科技股份有限公司 | Log event correlation analysis method and device capable of concurrent and interrupted analysis |
| US20110246826A1 (en) * | 2010-03-31 | 2011-10-06 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
| CN105608220A (en) * | 2016-01-08 | 2016-05-25 | 浪潮软件集团有限公司 | Acquisition method and device based on distributed message system |
| CN106534257A (en) * | 2016-09-29 | 2017-03-22 | 国家电网公司 | Multi-level cluster-type construction multi-source safety log collection system and method |
-
2019
- 2019-03-29 CN CN201910255935.7A patent/CN110011845B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110246826A1 (en) * | 2010-03-31 | 2011-10-06 | Cloudera, Inc. | Collecting and aggregating log data with fault tolerance |
| CN102158355A (en) * | 2011-03-11 | 2011-08-17 | 广州蓝科科技股份有限公司 | Log event correlation analysis method and device capable of concurrent and interrupted analysis |
| CN105608220A (en) * | 2016-01-08 | 2016-05-25 | 浪潮软件集团有限公司 | Acquisition method and device based on distributed message system |
| CN106534257A (en) * | 2016-09-29 | 2017-03-22 | 国家电网公司 | Multi-level cluster-type construction multi-source safety log collection system and method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114817187A (en) * | 2021-01-19 | 2022-07-29 | 成都鼎桥通信技术有限公司 | Log processing method and device |
| CN113111137A (en) * | 2021-04-30 | 2021-07-13 | 深圳壹账通智能科技有限公司 | Distributed system log real-time display method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110011845B (en) | 2022-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11379475B2 (en) | Analyzing tags associated with high-latency and error spans for instrumented software | |
| US11775501B2 (en) | Trace and span sampling and analysis for instrumented software | |
| US11409645B1 (en) | Intermittent failure metrics in technological processes | |
| Mayer et al. | An approach to extract the architecture of microservice-based software systems | |
| CN104809201B (en) | A kind of method and apparatus of database synchronization | |
| CN103235820B (en) | Date storage method and device in a kind of group system | |
| CN109558397B (en) | Data processing method, device, server and computer storage medium | |
| CN108228322B (en) | Distributed link tracking and analyzing method, server and global scheduler | |
| US11755531B1 (en) | System and method for storage of data utilizing a persistent queue | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| US11782920B1 (en) | Durable search queries for reliable distributed data retrieval | |
| CN110011845A (en) | Log collection method and system | |
| CN115622906A (en) | Application log capturing system and method | |
| CN114629949B (en) | Service monitoring method, electronic equipment and computer storage medium | |
| CN113422808B (en) | Internet of things platform HTTP information pushing method, system, device and medium | |
| CN109409948B (en) | Transaction abnormity detection method, device, equipment and computer readable storage medium | |
| CN113472858B (en) | Buried point data processing method and device and electronic equipment | |
| JP2008108154A (en) | Management system for operation performance information | |
| CN113297245A (en) | Method and device for acquiring execution information | |
| CN112579552A (en) | Log storage and calling method, device and system | |
| CN115525392A (en) | Container monitoring method and device, electronic equipment and storage medium | |
| US20230237049A1 (en) | Artifact life tracking storage | |
| CN112764988B (en) | Data segment acquisition method and device | |
| WO2022261249A1 (en) | Distributed task assignment, distributed alerts and supression management, and artifact life tracking storage in a cluster computing system | |
| CN112560938B (en) | Model training method and device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |