CN108984362A - Log collection method and device, storage medium, electronic equipment - Google Patents
Log collection method and device, storage medium, electronic equipment Download PDFInfo
- Publication number
- CN108984362A CN108984362A CN201710400067.8A CN201710400067A CN108984362A CN 108984362 A CN108984362 A CN 108984362A CN 201710400067 A CN201710400067 A CN 201710400067A CN 108984362 A CN108984362 A CN 108984362A
- Authority
- CN
- China
- Prior art keywords
- journal file
- log
- log collection
- collection method
- file
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3079—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by reporting only the changes of the monitored data
Abstract
The disclosure is directed to a kind of log collection method and device, storage medium, electronic equipment, which includes: the first journal file of monitoring and judges whether first journal file generates variation；When judging that first journal file generates variation, triggers the first journal file described in log collection programmed acquisition and generate the data of variation to obtain the second journal file；And by the second log file push into message queue so that the message queue completes output to second journal file.The real-time and availability of journal file can be improved in the disclosure.
Description
Technical field
This disclosure relates to which technical field of data processing, fills in particular to a kind of log collection method, log collection
It sets, computer readable storage medium and electronic equipment.
Background technique
In certain scientific & technical corporation, there can be many operation systems.Easy to use, the meeting in order to itself of each operation system
Different log frames, different log configurations and different log output formats and position are selected, but aforesaid way increases
The collection difficulty of log is added；Also, the log between program also can not related association analysis；It is also not easy to when program mal
Conjoint Analysis failure cause increases the work of system O&M.
It is by (such as can be 30 minutes or 1 every a preset time in a kind of existing log collection method
Hour etc.) actively acquire the once journal file in the preset time；But it is deposited by the journal file that this method obtains
In following problems: on the one hand, being unable to real-time monitoring journal file and acquired in real time when changing of journal file, drop
The low real-time and availability of journal file；It on the other hand, due to acquisition is all journal files in preset time,
So that including a large amount of redundant data in journal file, the delivery efficiency of journal file is reduced；In another aspect, output is a large amount of
It is useless repeat log, be not easy to log keynote message consult；In addition, log may write full disk under abnormal conditions, make
At more serious program mal and server problem.
It should be noted that information is only used for reinforcing the reason to the background of the disclosure disclosed in above-mentioned background technology part
Solution, therefore may include the information not constituted to the prior art known to persons of ordinary skill in the art.
Summary of the invention
The disclosure is designed to provide a kind of log collection method, log acquisition apparatus, computer readable storage medium
And electronic equipment, and then overcome caused by the limitation and defect due to the relevant technologies at least to a certain extent one or
Multiple problems.
According to one aspect of the disclosure, a kind of log collection method is provided, comprising:
It monitors the first journal file and judges whether first journal file generates variation；
When judging that first journal file generates variation, the first journal file described in log collection programmed acquisition is triggered
The data of variation are generated to obtain the second journal file；And
By the second log file push into message queue so that the message queue is completed to second log
The output of file.
In a kind of exemplary embodiment of the disclosure, before by the second log file push into message queue also
Include:
Second journal file is filtered and/or is cleaned.
In a kind of exemplary embodiment of the disclosure, second journal file is filtered includes:
Judge whether the data volume of second journal file reaches preset threshold；
When judging that the data volume of second journal file reaches the preset threshold, to second journal file into
Row filtering.
In a kind of exemplary embodiment of the disclosure, second journal file is filtered further include:
Calculate the data similarity of second journal file and judge the data similarity whether be greater than preset it is similar
Degree；
When judging that the data similarity is greater than the default similarity, the redundancy in second journal file is filtered
Set of metadata of similar data.
In a kind of exemplary embodiment of the disclosure, second journal file is filtered further include:
It calculates the time window weighted average of second journal file and judges the time window weighted average
Whether default average is greater than；
When judging that the time window weighted average are greater than the default average, second journal file is filtered
In redundancy time window data.
In a kind of exemplary embodiment of the disclosure, carrying out cleaning to second journal file includes:
To second journal file addition time identifier and log rank, the extraction pair in second journal file
The time identifier and log rank answered and one of denoising or a variety of is carried out to second journal file.
In a kind of exemplary embodiment of the disclosure, the log collection method further include:
Second journal file is obtained from the message queue and second journal file is subjected to external storage.
In a kind of exemplary embodiment of the disclosure, the log collection method further include:
Second journal file is obtained from the message queue and fault diagnosis is carried out to second journal file.
In a kind of exemplary embodiment of the disclosure, carrying out fault diagnosis to second journal file includes:
Statistic of classification is carried out to the data exception of second journal file；And
It alarms the disaster failover occurred in the data exception.
In a kind of exemplary embodiment of the disclosure, the log collection method further include:
Obtain third journal file and according to the data format of the third journal file to the third journal file into
Row parsing；And
Third journal file after parsing is pushed into the message queue.
According to one aspect of the disclosure, a kind of log acquisition apparatus is provided, comprising:
Journal file monitoring modular, for monitoring the first journal file and judging whether first journal file generates change
Change；
Journal file acquisition module, for triggering log collection journey when judging that first journal file generates variation
Sequence acquisition generates first journal file of variation to obtain the second journal file；
Journal file pushing module, for by the second log file push into message queue so that the message team
Column complete the output to second journal file.
According to one aspect of the disclosure, a kind of computer readable storage medium is provided, computer program is stored thereon with,
The computer program realizes log collection method described in above-mentioned any one when being executed by processor.
According to one aspect of the disclosure, a kind of electronic equipment is provided, comprising:
Processor；And
Memory, for storing the executable instruction of the processor；
Wherein, the processor is configured to execute day described in above-mentioned any one via the executable instruction is executed
Will acquisition method.
A kind of log collection method of the disclosure and device are generated by the first journal file of monitoring and in the first journal file
When variation, triggering the first journal file of log collection programmed acquisition generates the data of variation and obtains the second journal file, then
By the second log file push into message queue so that message queue completes the output to the second journal file；On the one hand, lead to
It crosses real-time monitoring journal file and is acquired in real time when changing of journal file, improve the real-time of journal file
And availability；On the other hand, changed journal file is only acquired, reduces the redundant data of journal file, improves
The efficiency of output.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, not
The disclosure can be limited.
Detailed description of the invention
The drawings herein are incorporated into the specification and forms part of this specification, and shows the implementation for meeting the disclosure
Example, and together with specification for explaining the principles of this disclosure.It should be evident that the accompanying drawings in the following description is only the disclosure
Some embodiments for those of ordinary skill in the art without creative efforts, can also basis
These attached drawings obtain other attached drawings.
Fig. 1 schematically shows a kind of flow chart of log collection method.
Fig. 2 schematically shows a kind of journal file exemplary diagram.
Fig. 3 schematically shows another journal file exemplary diagram.
Fig. 4 schematically shows a kind of flow chart of journal file filter method.
Fig. 5 schematically shows the flow chart of another journal file filter method.
Fig. 6 schematically shows the flow chart of another journal file filter method.
Fig. 7 schematically shows a kind of flow chart of journal file method for diagnosing faults.
Fig. 8 schematically shows a kind of block diagram of log acquisition apparatus.
Fig. 9 schematically shows a kind of electronic equipment exemplary diagram.
Specific embodiment
Example embodiment is described more fully with reference to the drawings.However, example embodiment can be with a variety of shapes
Formula is implemented, and is not understood as limited to example set forth herein；On the contrary, thesing embodiments are provided so that the disclosure will more
Fully and completely, and by the design of example embodiment comprehensively it is communicated to those skilled in the art.Described feature, knot
Structure or characteristic can be incorporated in any suitable manner in one or more embodiments.In the following description, it provides perhaps
More details fully understand embodiment of the present disclosure to provide.It will be appreciated, however, by one skilled in the art that can
It is omitted with technical solution of the disclosure one or more in the specific detail, or others side can be used
Method, constituent element, device, step etc..In other cases, be not shown in detail or describe known solution to avoid a presumptuous guest usurps the role of the host and
So that all aspects of this disclosure thicken.
In addition, attached drawing is only the schematic illustrations of the disclosure, it is not necessarily drawn to scale.Identical attached drawing mark in figure
Note indicates same or similar part, thus will omit repetition thereof.Some block diagrams shown in the drawings are function
Energy entity, not necessarily must be corresponding with physically or logically independent entity.These function can be realized using software form
Energy entity, or these functional entitys are realized in one or more hardware modules or integrated circuit, or at heterogeneous networks and/or place
These functional entitys are realized in reason device device and/or microcontroller device.
A kind of log collection method is provided firstly in this example embodiment.Refering to what is shown in Fig. 1, the log collection method
It may comprise steps of:
Step S110. monitors the first journal file and judges whether first journal file generates variation.
Step S120. triggers described in log collection programmed acquisition the when judging that first journal file generates variation
One journal file generates the data of variation to obtain the second journal file.
Step S130. is by the second log file push into message queue so that the message queue is completed to described
The output of second journal file.
In above-mentioned log collection method, on the one hand, by real-time monitoring journal file and in changing of journal file
When acquired in real time, improve the real-time and availability of journal file；On the other hand, changed day is only acquired
Will file reduces the redundant data of journal file, improves the efficiency of output.
In the following, by each step of log collection method above-mentioned in this example embodiment is described in detail.
In step s 110, it monitors the first journal file and judges whether first journal file generates variation.
In this example embodiment, above-mentioned journal file may include Log Names, log source, event id, rank,
Time, task category and keyword etc. are recorded, can also be deposited including journal file maximum backup quantity, journal file maximum
Amplify small, journal file BACKUP TIME interval, journal file storage path, whether be shown in console and whether using debugging
Log etc., there is no special restriction on this for this example embodiment；Whether above-mentioned journal file, which generates variation, is construed as
Whether the rank of journal file generates variation, and whether the source for being also construed as journal file generates variation or log text
Whether the title of part changes, and there is no special restriction on this for this example embodiment；The rank of above-mentioned journal file can wrap
Include debugging, information, warning and mistake, also may include it is serious etc., this example embodiment is not particularly limited this.
Wherein, debugging, which can be defined as, points out fine granularity message event, and debugging is very helpful to debugging utility；Information
It can be defined as the operational process for showing that message highlights application program on coarse grain level；Warning may indicate that and can go out
The situation of existing latent fault；Although mistake can be defined as point out that error event occurs, but still does not influence the continuation of system
Operation；Serious error, which can be defined as each grave error event, will will lead to exiting for application program.In this example
Monitoring refers to application log file shown in Fig. 2, and judges whether the application log file changes (log rank
It changes and/or log source changes).It, can also be in addition, in other illustrative embodiments of the disclosure
The title of journal file changes or records time change to judge whether journal file changes, this example embodiment is to this
It does not do specifically limited.
In the step s 120, when judging that first journal file generates variation, log collection programmed acquisition institute is triggered
It states the first journal file and generates the data of variation to obtain the second journal file.For example:
With continued reference to shown in Fig. 2, when the rank for monitoring journal file changes and (becomes mistake from information), trigger
Log collection programmed acquisition generates the data of variation and obtains above-mentioned second journal file (can be with reference to shown in Fig. 3).In addition,
In other illustrative embodiments of the disclosure, above-mentioned second journal file is also possible to monitor that the source of journal file occurs
The data that the generation variation of log collection programmed acquisition is triggered when variation obtain, and there is no special restriction on this for this example embodiment.
In step s 130, by the second log file push into message queue so that the message queue complete pair
The output of second journal file.
In this example embodiment, message queue can be defined as the container that message is saved in message transmitting procedure,
Can solve using coupling, asynchronous message and the problems such as flow peak clipping, for example, may include ActiveMQ, RabbitMQ or
ZeroMQ etc.；In this example, message queue can be used for storing above-mentioned second journal file, and in above-mentioned second log text
When part carries out other processing, the output to above-mentioned second journal file is completed.For example:
It will be with reference to above-mentioned second journal file active push shown in Fig. 3 into message queue ActiveMQ, when needs pair
When above-mentioned second journal file carries out external storage or fault diagnosis, can obtain directly from message queue ActiveMQ should
Second journal file；Further, by above-mentioned second journal file active push into message queue, is improved this second
The timeliness of journal file, and ensure that the express statistic of accident analysis.
In this example embodiment, the daily record data without demur of big data quantity is avoided to occupy more resources, while from being also
The memory space of the above-mentioned message queue of saving can avoid journey by the way of active filtering and/cleaning redundant logs data
Sequence failure brings a large amount of abnormal data；Specifically, can be with before by above-mentioned second log file push into message queue
First above-mentioned second journal file is filtered and/or is cleaned.Wherein, refering to what is shown in Fig. 4, being carried out to above-mentioned second journal file
Filtering may comprise steps of:
In step S410, judge whether the data volume of second journal file reaches preset threshold.
In this example embodiment, in order to avoid filtering function bring resource overhead, first to above-mentioned second log text
The data volume of part is counted, and filtering function can be opened by then judging whether the data volume of above-mentioned second journal file reaches
(preset threshold can use collected historical data and analyzed to obtain preset threshold, such as can be 200MB, 500MB
Or 1GB etc., there is no special restriction on this for this example embodiment).In addition, in other illustrative embodiments of the disclosure
In, filtering function bring resource overhead can also be avoided in other manners, this example embodiment does not do special limit to this
System.
In the step s 420, when judging that the data volume of second journal file reaches the preset threshold, to described
Second journal file is filtered.Refering to what is shown in Fig. 5, the data volume when above-mentioned second journal file reaches above-mentioned preset threshold
When, being filtered to above-mentioned second journal file may include step 4202 and step S4204.Wherein:
In step S4202, calculates the data similarity of second journal file and judge that the data similarity is
It is no to be greater than default similarity.For example:
It can be with continued reference to similarity (such as the rank that the data that each journal file includes shown in Fig. 2, can be calculated separately
Similarity, the similarity of date and time, the similarity in source etc.), then calculate the average value of each similarity, and judge
The average value whether be greater than default similarity (the default similarity can use collected historical data and be analyzed to obtain,
Such as can be 0.7,0.75 or 0.8 etc., this example embodiment does not sit this specifically limited).In addition, in the disclosure
In other illustrative embodiments, above-mentioned data similarity can also be calculated in other manners, this example embodiment is to this
It does not do specifically limited.
In step S4204, when judging that the data similarity is greater than the default similarity, the second day is filtered
Redundancy set of metadata of similar data in will file.For example:
The average value for calculating above-mentioned each similarity is 0.9, is greater than above-mentioned default similarity 0.8, then filters out each similarity
Redundant data greater than 0.9 retains wherein any a.
In this exemplary another embodiment, refering to what is shown in Fig. 6, the data volume when above-mentioned second journal file reaches
When above-mentioned preset threshold, being filtered to above-mentioned second journal file may include step 4206 and step S4208.Wherein:
In step S4206, calculates the time window weighted average of second journal file and judge the time
Whether window weight average is greater than default average.
In this example embodiment, above-mentioned time window is construed as the date and time data of each journal file；
Above-mentioned time window weighted average can be calculated by the number that the date and time using each journal file occurs.?
In this example, with continued reference to shown in Fig. 2, the weighted average of the date and time of above-mentioned second journal file is calculated, are then sentenced
Whether the weighted average that break are greater than default average (the default average can use collected historical data and be analyzed
It obtains, such as can be with 2.78,5.03 or 10.50 etc., there is no special restriction on this for this example embodiment).
In step S4208, when judging that the time window weighted average are greater than the default average, institute is filtered
State the redundancy time window data in the second journal file.For example:
Calculating above-mentioned weighted average is 6.88, is greater than above-mentioned default default average 5.03, then filter out each date and
The weight of time is greater than 6.88 redundant data, retains wherein any a.
Further, carrying out cleaning to above-mentioned second journal file may include: when adding to second journal file
Between identify and log rank, extract in second journal file corresponding time identifier and log rank and to institute
It states the second journal file and carries out one of denoising or a variety of.Specifically:
In order to which the data of each journal file can be carried out to more arrange comprehensively, so that higher for several log ranks
The available priority processing of journal file, can first judge in the data of each journal file whether comprising time identifier (date with
And the time) and log rank, when in the data for judging each journal file comprising time identifier and log rank, extracting should
Event identifier and log rank；It, can when not including time identifier and log rank in the data for judging each journal file
To add (automatic addition or carrying out manual intervention) time identifier and logging level in the data of each journal file；Into one
Step, it is more succinct in order to allow each journal file, and also to the burden of message queue is reduced, it can also be to each log
File carries out denoising, etc..In addition, in other illustrative embodiments of the disclosure, it can also be in other manners
Each journal file is cleaned, there is no special restriction on this for this example embodiment.
The disclosure additionally provides another log collection method, which can also include: from the message
Second journal file is obtained in queue and second journal file is subjected to external storage；And from the message queue
It is middle to obtain second journal file and fault diagnosis is carried out to second journal file.Specifically:
In order to facilitate obtain history log file data and reduce message queue burden, can by each journal file into
Row external storage, such as can be database purchase either external distributed file storage etc.；Further, in order to can be with
The problem is timely solved, fault diagnosis can also be carried out to each journal file；Wherein, refering to what is shown in Fig. 7, fault diagnosis
It may include step S710 and step S720.Wherein:
In step S710, statistic of classification is carried out to the data exception of second journal file.
In this example embodiment, pass through the data exception memory statistic of classification to each journal file and provide and history
The year-on-year and ring of data can check problem for application system and provide reference than report.In addition, other in the disclosure are exemplary
In embodiment, problem can also be checked in other manners for application system and reference is provided, this example embodiment to this not
It does specifically limited.
In step S720, alarm the disaster failover occurred in the data exception.
It, can be according to the keyword in the data exception of each journal file, disaster failover, weight in this example embodiment
Situations such as big abnormal, alarms, then key log is sent to the person liable of application system by SMS platform and mail；
So that the available quick solution of problem.
Further, the log collection method that the disclosure provides can also carry out real-time logs file to each journal file and push away
It send and offline logs file polling；Also, the condition of journal file can also be provided for application system using data base tool
Inquiry, conjunctive query carry out fault diagnosis convenient for application system, and program health status is analyzed and abnormal cause is checked etc.,
And then realize effect analysis, Attack monitoring, business interface condition monitoring and service security monitoring etc..
This example embodiment additionally provides another journal file acquisition method, which can also wrap
It includes: obtaining third journal file and the third journal file is solved according to the data format of the third journal file
Analysis；Then the third journal file after parsing is pushed into the message queue.Specifically:
Third-party journal file can be obtained from database either external distributed file, then according to the third
The data format and configuration log format of the journal file of side parse the third-party journal file, then will parsing
Journal file afterwards pushes in message queue to carry out subsequent step (fault diagnosis etc.)；By the above-mentioned means, can be with
Solve the aggregation problem of multisystem log, the failure screening that O&M efficiency can be improved, while being also convenient between multisystem.
In addition, although describing each step of method in the disclosure in the accompanying drawings with particular order, this does not really want
These steps must be executed in this particular order by asking or implying, or having to carry out step shown in whole could realize
Desired result.Additional or alternative, it is convenient to omit multiple steps are merged into a step and executed by certain steps, and/
Or a step is decomposed into execution of multiple steps etc..
This example embodiment additionally provides a kind of log acquisition apparatus.Refering to what is shown in Fig. 8, the log acquisition apparatus can be with
Including journal file monitoring modular 810, journal file acquisition module 820 and journal file pushing module 830.Wherein:
Journal file monitoring modular 810 can be used for monitoring the first journal file and whether judge first journal file
Generate variation.
Journal file acquisition module 820 can be used for triggering log when judging that first journal file generates variation
Capture program acquisition generates first journal file of variation to obtain the second journal file.
Journal file pushing module 830 can be used for the second log file push into message queue so that described
Message queue completes the output to second journal file.
The detail of each module has carried out in corresponding log collection method in detail in above-mentioned log acquisition apparatus
Description is considered carefully, therefore details are not described herein again.
It should be noted that although being referred to several modules or list for acting the equipment executed in the above detailed description
Member, but this division is not enforceable.In fact, according to embodiment of the present disclosure, it is above-described two or more
Module or the feature and function of unit can embody in a module or unit.Conversely, an above-described mould
The feature and function of block or unit can be to be embodied by multiple modules or unit with further division.
It also proposed a kind of non-transient computer readable storage medium in this example embodiment, be stored thereon with computer
Program, the computer program realize above-mentioned palletizing method when being executed by processor.
Computer readable storage medium may include in a base band or as carrier wave a part propagate data-signal,
In carry readable program code.The data-signal of this propagation can take various forms, including but not limited to electromagnetic signal,
Optical signal or above-mentioned any appropriate combination.Computer readable storage medium can send, propagates or transmit for by referring to
Enable execution system, device or device use or program in connection.
The program code for including in computer readable storage medium can transmit with any suitable medium, including but unlimited
In wireless, wired, optical cable, radio frequency etc. or above-mentioned any appropriate combination.
It also proposed a kind of electronic equipment in this example embodiment, as shown in figure 9, the electronic equipment 10 includes: processing
Component 11 can further include one or more processors, and the memory resource as representated by memory 12, be used for
Store the instruction that can be executed by processing component 11, such as application program.The application program stored in memory 12 may include one
It is a or it is more than one each correspond to one group of instruction module.In addition, processing component 11 is configured as executing instruction, to hold
The row above method.
The electronic equipment 10 can also include: a power supply module, and power supply module is configured to execution electronic equipment 10
Carry out power management；One wired or wireless network interface 13 is configured to electronic equipment 10 being connected to network；And one
Input and output (I/O) interface 14.The electronic equipment 10 can be operated based on the operating system for being stored in memory 12, such as
Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM or similar.
Through the above description of the embodiments, those skilled in the art is it can be readily appreciated that example described herein is implemented
Mode can also be realized by software realization in such a way that software is in conjunction with necessary hardware.Therefore, according to the disclosure
The technical solution of embodiment can be embodied in the form of software products, which can store non-volatile at one
Property storage medium (can be CD-ROM, USB flash disk, mobile hard disk etc.) in or network on, including some instructions are so that a calculating
Equipment (can be personal computer, server, mobile terminal or network equipment etc.) is executed according to disclosure embodiment
Method.
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to its of the disclosure
Its embodiment.This application is intended to cover any variations, uses, or adaptations of the disclosure, these modifications, purposes or
Person's adaptive change follows the general principles of this disclosure and including the undocumented common knowledge in the art of the disclosure
Or conventional techniques.The description and examples are only to be considered as illustrative, and the true scope and spirit of the disclosure are by appended
Claim is pointed out.
Claims (13)
1. a kind of log collection method characterized by comprising
It monitors the first journal file and judges whether first journal file generates variation；
When judging that first journal file generates variation, triggers the first journal file described in log collection programmed acquisition and generate
The data of variation are to obtain the second journal file；And
By the second log file push into message queue so that the message queue is completed to second journal file
Output.
2. log collection method according to claim 1, which is characterized in that by the second log file push to disappearing
Before ceasing in queue further include:
Second journal file is filtered and/or is cleaned.
3. log collection method according to claim 2, which is characterized in that be filtered packet to second journal file
It includes:
Judge whether the data volume of second journal file reaches preset threshold；
When judging that the data volume of second journal file reaches the preset threshold, second journal file was carried out
Filter.
4. log collection method according to claim 3, which is characterized in that be filtered also to second journal file
Include:
It calculates the data similarity of second journal file and judges whether the data similarity is greater than default similarity；
When judging that the data similarity is greater than the default similarity, the redundancy filtered in second journal file is similar
Data.
5. log collection method according to claim 3, which is characterized in that be filtered also to second journal file
Include:
It calculates the time window weighted average of second journal file and whether judges the time window weighted average
Greater than default average；
When judging that the time window weighted average are greater than the default average, filter in second journal file
Redundancy time window data.
6. log collection method according to claim 3, which is characterized in that carry out cleaning packet to second journal file
It includes:
It is extracted to second journal file addition time identifier and log rank, in second journal file corresponding
Time identifier and log rank and one of denoising or a variety of is carried out to second journal file.
7. log collection method according to claim 1, which is characterized in that the log collection method further include:
Second journal file is obtained from the message queue and second journal file is subjected to external storage.
8. log collection method according to claim 1, which is characterized in that the log collection method further include:
Second journal file is obtained from the message queue and fault diagnosis is carried out to second journal file.
9. log collection method according to claim 8, which is characterized in that carry out failure to second journal file and examine
It is disconnected to include:
Statistic of classification is carried out to the data exception of second journal file；And
It alarms the disaster failover occurred in the data exception.
10. log collection method according to claim 1, which is characterized in that the log collection method further include:
It obtains third journal file and the third journal file is solved according to the data format of the third journal file
Analysis；And
Third journal file after parsing is pushed into the message queue.
11. a kind of log acquisition apparatus characterized by comprising
Journal file monitoring modular, for monitoring the first journal file and judging whether first journal file generates variation；
Journal file acquisition module, for when judging that first journal file generates variation, triggering log collection program to be adopted
Collection generates first journal file of variation to obtain the second journal file；
Journal file pushing module, for by the second log file push into message queue so that the message queue is complete
The output of pairs of second journal file.
12. a kind of computer readable storage medium, is stored thereon with computer program, which is characterized in that the computer program
Claim 1-10 described in any item log collection methods are realized when being executed by processor.
13. a kind of electronic equipment characterized by comprising
Processor；And
Memory, for storing the executable instruction of the processor；
Wherein, the processor is configured to require 1-10 described in any item via executing the executable instruction and carry out perform claim
Log collection method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710400067.8A CN108984362A (en) | 2017-05-31 | 2017-05-31 | Log collection method and device, storage medium, electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710400067.8A CN108984362A (en) | 2017-05-31 | 2017-05-31 | Log collection method and device, storage medium, electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108984362A true CN108984362A (en) | 2018-12-11 |
Family
ID=64502146
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710400067.8A Pending CN108984362A (en) | 2017-05-31 | 2017-05-31 | Log collection method and device, storage medium, electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108984362A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110888783A (en) * | 2019-11-21 | 2020-03-17 | 望海康信(北京)科技股份公司 | Monitoring method and device of micro-service system and electronic equipment |
| CN111324505A (en) * | 2019-11-12 | 2020-06-23 | 杭州海康威视系统技术有限公司 | Log collection method and device, electronic equipment and storage medium |
| US11392480B2 (en) * | 2019-12-19 | 2022-07-19 | Jpmorgan Chase Bank, N.A. | System and method for implementing a log file processing module |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101325520A (en) * | 2008-06-17 | 2008-12-17 | 南京邮电大学 | Method for locating and analyzing fault of intelligent self-adapting network based on log |
| CN101950293A (en) * | 2010-08-11 | 2011-01-19 | 东软集团股份有限公司 | Log extraction method and device |
| US20130227352A1 (en) * | 2012-02-24 | 2013-08-29 | Commvault Systems, Inc. | Log monitoring |
| CN103838659A (en) * | 2014-02-17 | 2014-06-04 | 大唐移动通信设备有限公司 | Method and device for controlling system logs |
| CN105574096A (en) * | 2015-12-10 | 2016-05-11 | 惠州Tcl移动通信有限公司 | Method and system for obtaining, uploading and analyzing log information |
| CN105893208A (en) * | 2016-03-31 | 2016-08-24 | 城云科技（杭州）有限公司 | Cloud computing platform system fault prediction method based on hidden semi-Markov models |
| CN106156355A (en) * | 2016-07-27 | 2016-11-23 | 腾讯科技（深圳）有限公司 | Log processing method and device |
-
2017
- 2017-05-31 CN CN201710400067.8A patent/CN108984362A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101325520A (en) * | 2008-06-17 | 2008-12-17 | 南京邮电大学 | Method for locating and analyzing fault of intelligent self-adapting network based on log |
| CN101950293A (en) * | 2010-08-11 | 2011-01-19 | 东软集团股份有限公司 | Log extraction method and device |
| US20130227352A1 (en) * | 2012-02-24 | 2013-08-29 | Commvault Systems, Inc. | Log monitoring |
| CN103838659A (en) * | 2014-02-17 | 2014-06-04 | 大唐移动通信设备有限公司 | Method and device for controlling system logs |
| CN105574096A (en) * | 2015-12-10 | 2016-05-11 | 惠州Tcl移动通信有限公司 | Method and system for obtaining, uploading and analyzing log information |
| CN105893208A (en) * | 2016-03-31 | 2016-08-24 | 城云科技（杭州）有限公司 | Cloud computing platform system fault prediction method based on hidden semi-Markov models |
| CN106156355A (en) * | 2016-07-27 | 2016-11-23 | 腾讯科技（深圳）有限公司 | Log processing method and device |
Non-Patent Citations (1)
| Title |
|---|
| 武剑: "《内部评级理论、方法与实务 巴塞尔新资本协议核心技术》", 31 July 2007 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111324505A (en) * | 2019-11-12 | 2020-06-23 | 杭州海康威视系统技术有限公司 | Log collection method and device, electronic equipment and storage medium |
| CN110888783A (en) * | 2019-11-21 | 2020-03-17 | 望海康信(北京)科技股份公司 | Monitoring method and device of micro-service system and electronic equipment |
| CN110888783B (en) * | 2019-11-21 | 2023-07-07 | 望海康信(北京)科技股份公司 | Method and device for monitoring micro-service system and electronic equipment |
| US11392480B2 (en) * | 2019-12-19 | 2022-07-19 | Jpmorgan Chase Bank, N.A. | System and method for implementing a log file processing module |
| US11720473B2 (en) | 2019-12-19 | 2023-08-08 | Jpmorgan Chase Bank, N.A. | System and method for implementing a log file processing module |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110661659B (en) | Alarm method, device and system and electronic equipment | |
| CN105488610B (en) | Fault real-time analysis and diagnosis method for power application system | |
| CN105159964B (en) | A kind of log monitoring method and system | |
| CN103401698B (en) | For the monitoring system that server health is reported to the police in server set group operatione | |
| CN101997709B (en) | Root alarm data analysis method and system | |
| US11093349B2 (en) | System and method for reactive log spooling | |
| CN111046011B (en) | Log collection method, system, device, electronic equipment and readable storage medium | |
| CN109977089A (en) | Blog management method, device, computer equipment and computer readable storage medium | |
| CN110362455B (en) | Data processing method and data processing device | |
| US7610172B2 (en) | Method and system for monitoring non-occurring events | |
| CN111881011A (en) | Log management method, platform, server and storage medium | |
| CN112311617A (en) | Configured data monitoring and alarming method and system | |
| CN108509326A (en) | A kind of service state statistical method and system based on nginx daily records | |
| CN111782486A (en) | Alarm implementation method and system based on dynamic configuration | |
| CN107544832A (en) | A kind of monitoring method, the device and system of virtual machine process | |
| CN109800098A (en) | Service exception node positioning method, device, computer equipment and storage medium | |
| US20200372372A1 (en) | Predicting the disaster recovery invocation response time | |
| CN108984362A (en) | Log collection method and device, storage medium, electronic equipment | |
| CN112395156A (en) | Fault warning method and device, storage medium and electronic equipment | |
| US20200099570A1 (en) | Cross-domain topological alarm suppression | |
| Zhou et al. | Recommending ticket resolution using feature adaptation | |
| US9443196B1 (en) | Method and apparatus for problem analysis using a causal map | |
| CN116489005A (en) | Log service system and log processing method | |
| CN112882892B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN110677271B (en) | Big data alarm method, device, equipment and storage medium based on ELK |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181211 |