CN110413478A - A kind of method, equipment and medium monitoring log processing - Google Patents
A kind of method, equipment and medium monitoring log processing Download PDFInfo
- Publication number
- CN110413478A CN110413478A CN201910578079.9A CN201910578079A CN110413478A CN 110413478 A CN110413478 A CN 110413478A CN 201910578079 A CN201910578079 A CN 201910578079A CN 110413478 A CN110413478 A CN 110413478A
- Authority
- CN
- China
- Prior art keywords
- log
- file server
- duplicate removal
- classification
- collect
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000012544 monitoring process Methods 0.000 title claims abstract description 18
- 238000012545 processing Methods 0.000 title claims abstract description 12
- 238000001914 filtration Methods 0.000 claims abstract description 12
- 230000004044 response Effects 0.000 claims abstract description 8
- 238000004590 computer program Methods 0.000 claims description 15
- 230000006870 function Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides a kind of method, equipment and medium for monitoring log processing, this method includes that the log for being collected monitor supervision platform based on time dimension carries out duplicate removal and classification；Duplicate removal and sorted log are filtered based on default filtering configuration and are sent to file server；In response to presetting characteristic events, the log in file server is inquired in advance and is collected log.By using method, apparatus of the invention, equipment and medium, monitoring log can be effectively handled, accelerates log writing speed, provides and flexibly exports and collect configuration, accelerates inquiry velocity.
Description
Technical field
This field is related to computer field, and relates more specifically to a kind of method, apparatus, equipment for monitoring log processing
And medium.
Background technique
The features such as monitoring log in monitor supervision platform has quantity more, and similarity is high.On the one hand, log sources are more,
It contains much information；On the other hand, in short time interval, even the same time, there may be similar log informations.Before massive logs
It puts, how effectively to store and inquiry log, be a problem problem to be solved.
A kind of scheme stores log using the time as dimension with full dose in real time at present, and user can check certain that oneself is needed
The log information of one period.In this scheme, to storage capacity requirement height, user's operation is cumbersome, needs to download decompression log
File is searched manually.
Summary of the invention
In view of this, the purpose of the embodiment of the present invention is to propose a kind of method, equipment and Jie for monitoring log processing
Matter, this method can effectively handle monitoring log, accelerate log writing speed, provide and flexibly export and collect configuration, add
Fast inquiry velocity.
Based on above-mentioned purpose, the one aspect of the embodiment of the present invention provides a kind of method for monitoring log processing, packet
Include following steps:
Duplicate removal and classification are carried out based on the log that time dimension collects monitor supervision platform；
Duplicate removal and sorted log are filtered based on default filtering configuration and are sent to file server；
In response to presetting characteristic events, the log in file server is inquired in advance and is collected log.
According to one embodiment of present invention, duplicate removal and classification are carried out based on the log that time dimension collects monitor supervision platform
It include: that duplicate removal and classification are carried out based on the keyword in the same period.
According to one embodiment of present invention, the period includes 1 to 5 second.
According to one embodiment of present invention, it is configured based on default filtering and duplicate removal and sorted log is filtered and sent
It include: to be filtered based on the file type in default filtering configuration to file server.
It according to one embodiment of present invention, further include sending browser, client and mailbox for filtered log.
According to one embodiment of present invention, in response to presetting characteristic events, the log in file server is carried out pre-
Inquiring and collect by log includes: to be looked into based on failure keyword of the server failure to the log in file server
It askes and caching is provided.
According to one embodiment of present invention, log collect includes: to be returned based on time and capacity to log
Collection.
According to one embodiment of present invention, it collects including being split and compressing to log.
The other side of the embodiment of the present invention additionally provides a kind of computer equipment, comprising:
At least one processor；And
Memory, memory are stored with the computer program that can be run on a processor, and processor executes when executing program
The above-mentioned method of any one.
The other side of the embodiment of the present invention additionally provides a kind of computer readable storage medium, computer-readable
Storage medium is stored with computer program, and any one above-mentioned method is executed when computer program is executed by processor.
The present invention has following advantageous effects: the method for monitoring log processing provided in an embodiment of the present invention passes through
Duplicate removal and classification are carried out based on the log that time dimension collects monitor supervision platform；It will be after duplicate removal and classification based on default filtering configuration
Log filter and be sent to file server；In response to presetting characteristic events, the log in file server is looked into advance
The technical solution ask and collect log can effectively handle monitoring log, accelerate log writing speed, provide flexibly
Output and collect configuration, accelerate inquiry velocity.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
Other embodiments are obtained according to these attached drawings.
Fig. 1 is the schematic flow chart according to the method for the monitoring log processing of one embodiment of the invention；
Fig. 2 is the schematic block diagram according to the device of the monitoring log processing of another embodiment of the present invention.
Specific embodiment
To make the objectives, technical solutions, and advantages of the present invention clearer, below in conjunction with specific embodiment, and reference
The embodiment of the present invention is further described in attached drawing.
Based on above-mentioned purpose, the first aspect of the embodiment of the present invention proposes a kind of method for monitoring log processing
One embodiment.Shown in fig. 1 is the schematic flow chart of this method.
As shown in fig. 1, this method may comprise steps of:
S1 carries out duplicate removal and classification based on the log that time dimension collects monitor supervision platform；
S2 is based on default filtering configuration and filters duplicate removal and sorted log and be sent to file server；
Log in file server is inquired in advance in response to presetting characteristic events and is collected log by S3.
By above technical scheme, monitoring log can be effectively handled, accelerates log writing speed, is provided flexible defeated
Out and configuration is collected, accelerates inquiry velocity.
In a preferred embodiment of the invention, the log collected monitor supervision platform based on time dimension carry out duplicate removal and
Classification includes: to carry out duplicate removal and classification based on the keyword in the same period.In a preferred embodiment of the invention, when
Between section include 1 to 5 second.It can choose the period according to the size of data volume in log, the data volume big time is then short, and data volume is small
Time is then grown, and carries out duplicate removal and classification according to keyword within the selected period.
In a preferred embodiment of the invention, duplicate removal and sorted log are filtered simultaneously based on default filtering configuration
Being sent to file server includes: to be filtered based on the file type in default filtering configuration.There are many files classes in log
Type, the log of some file types be it is unwanted, as the case may be can it is configurating filtered fall file type, to reduce day
The amount of storage of will.
In a preferred embodiment of the invention, further include send filtered log to browser, client and
Mailbox.Filtered log, which can according to need, is sent to different receiving ends, further includes this paper others not mentioned receiving end.
In a preferred embodiment of the invention, in response to preset characteristic events, by the log in file server into
Row is pre- inquire and log collect include: based on server failure to the failure keyword of the log in file server into
Row is inquired and provides caching.When server failure, the keyword that reports an error can be based in the log being stored in file server
Automatically inquired and cached, can be directly displayed out as engineer or user search log it is inquiring in advance as a result,
Accelerate the speed of inquiry.
In a preferred embodiment of the invention, log collect include: based on time and capacity to log into
Row collects.In a preferred embodiment of the invention, it collects including being split and compressing to log.It can be periodically to day
Will is split and compresses, such as daily log is put into a file and is compressed；It can also quantify to log
It is split and compresses, such as log often reaches 1G and is just stored in a file and compressed.
It should be noted that those of ordinary skill in the art will appreciate that realizing the whole in above-described embodiment method or portion
Split flow can instruct related hardware to complete by computer program, and above-mentioned program can be stored in computer-readable
In storage medium, the program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein, storage medium can
For magnetic disk, CD, read-only memory (Read-Only Memory, ROM) or random access memory (Random Access
Memory, RAM) etc..It is identical to can achieve corresponding aforementioned any means embodiment for the embodiment of above-mentioned computer program
Or similar effect.
In addition, disclosed method is also implemented as the computer program executed by CPU according to embodiments of the present invention, it should
Computer program may be stored in a computer readable storage medium.When the computer program is executed by CPU, the present invention is executed
The above-mentioned function of being limited in method disclosed in embodiment.
Based on above-mentioned purpose, the second aspect of the embodiment of the present invention proposes a kind of device for monitoring log processing,
Fig. 2 shows the schematic block diagrams of device 200, comprising:
Preprocessor 201, the log that preprocessor 201 is configured to collect monitor supervision platform is based on the key in certain time
Word carries out duplicate removal and classification；
Output interface 202, output interface 202 are connected to preprocessor 201 and are configured to default filtering configuration and will go
Weight and sorted log are filtered based on file type, and filtered log is transmitted；
File server 203, file server 203 are connected to output interface 202 and are configured to receive the biography of output interface 202
Defeated log；
Requestor 204, requestor 204 are connected to file server 203 and are configured that based on server failure in advance in text
Keyword is inquired in part server 203 and caching is provided；
Collect device 205, collect device 205 be connected to file server 203 and be configured in file server 203 based on when
Between and capacity log is split and is compressed.
Based on above-mentioned purpose, the third aspect of the embodiment of the present invention proposes a kind of computer equipment, comprising:
At least one processor；And
Memory, memory are stored with the computer program that can be run on a processor, and processor executes when executing program
The above-mentioned method of any one.
Based on above-mentioned purpose, the 4th aspect of the embodiment of the present invention proposes a kind of computer readable storage medium,
Computer-readable recording medium storage has computer program, and it is above-mentioned that any one is executed when computer program is executed by processor
Method.
By using method of the invention, it is possible to effectively handle monitoring log, accelerates log writing speed, provide flexibly
Output and collect configuration, accelerate inquiry velocity.
It is important to note that the embodiment of above system uses the embodiment of the above method to illustrate each mould
The course of work of block, those skilled in the art can be it is readily conceivable that other by these module applications to the above method be implemented
In example.
In addition, above method step and system unit or module also can use controller and for storing so that controlling
Device processed realizes that the computer readable storage medium of above-mentioned steps or unit or the computer program of functions of modules is realized.
Those skilled in the art will also understand is that, various illustrative logical blocks, mould in conjunction with described in disclosure herein
Block, circuit and algorithm steps may be implemented as the combination of electronic hardware, computer software or both.It is hard in order to clearly demonstrate
This interchangeability of part and software, with regard to various exemplary components, square, module, circuit and step function to its into
General description is gone.This function is implemented as software and is also implemented as hardware depending on concrete application and application
To the design constraint of whole system.The function that those skilled in the art can realize in various ways for every kind of concrete application
Can, but this realization decision should not be interpreted as causing a departure from range disclosed by the embodiments of the present invention.
Above-described embodiment, especially any " preferably " embodiment are the possibility examples realized, and only for clear geography
It solves the principle of the present invention and proposes.It can be in the case where not departing from the spirit and principle of techniques described herein to above-mentioned reality
It applies example and carries out many change and modification.All modifications are intended to be included in the scope of the present disclosure and are protected by appended claims
Shield.
Claims (10)
1. a kind of method for monitoring log processing, which comprises the following steps:
Duplicate removal and classification are carried out based on the log that time dimension collects monitor supervision platform；
Duplicate removal and the sorted log are filtered based on default filtering configuration and are sent to file server；
In response to presetting characteristic events, the log in file server is inquired in advance and is returned the log
Collection.
2. the method according to claim 1, wherein being carried out based on the log that time dimension collects monitor supervision platform
Duplicate removal and classification include: to carry out the duplicate removal and classification based on the keyword in the same period.
3. according to the method described in claim 2, it is characterized in that, the period includes 1 to 5 second.
4. the method according to claim 1, wherein based on default filtering configuration by duplicate removal and sorted described
It includes: to be filtered based on the file type in the default filtering configuration that log, which filters and is sent to file server,.
5. the method according to claim 1, wherein further include send the filtered log to it is clear
Look at device, client and mailbox.
6. the method according to claim 1, wherein in response to preset characteristic events, will be in file server
It includes: based on server failure to described in file server that the log, which is inquired in advance and collect by the log,
The failure keyword of log is inquired and provides caching.
7. the method according to claim 1, wherein it includes: based on time and appearance that the log, which collect,
Amount is collected to described in log progress.
8. the method according to claim 1, wherein described collect including the log is split and is pressed
Contracting.
9. a kind of computer equipment, comprising:
At least one processor；And
Memory, the memory are stored with the computer program that can be run on the processor, which is characterized in that the place
Manage the method executed as described in claim 1-7 any one when device executes described program.
10. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In perform claim requires method described in 1-7 any one when the computer program is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910578079.9A CN110413478A (en) | 2019-06-28 | 2019-06-28 | A kind of method, equipment and medium monitoring log processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910578079.9A CN110413478A (en) | 2019-06-28 | 2019-06-28 | A kind of method, equipment and medium monitoring log processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110413478A true CN110413478A (en) | 2019-11-05 |
Family
ID=68358531
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910578079.9A Withdrawn CN110413478A (en) | 2019-06-28 | 2019-06-28 | A kind of method, equipment and medium monitoring log processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110413478A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111639059A (en) * | 2020-05-28 | 2020-09-08 | 深圳壹账通智能科技有限公司 | Log information storage and positioning method, electronic equipment and storage medium |
| CN112800016A (en) * | 2020-12-31 | 2021-05-14 | 武汉思普崚技术有限公司 | Log data classification and sorting method and device |
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN114449628A (en) * | 2021-12-30 | 2022-05-06 | 荣耀终端有限公司 | Log data processing method, electronic device and medium thereof |
-
2019
- 2019-06-28 CN CN201910578079.9A patent/CN110413478A/en not_active Withdrawn
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN112866319B (en) * | 2019-11-28 | 2023-10-13 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN111639059A (en) * | 2020-05-28 | 2020-09-08 | 深圳壹账通智能科技有限公司 | Log information storage and positioning method, electronic equipment and storage medium |
| CN112800016A (en) * | 2020-12-31 | 2021-05-14 | 武汉思普崚技术有限公司 | Log data classification and sorting method and device |
| CN114449628A (en) * | 2021-12-30 | 2022-05-06 | 荣耀终端有限公司 | Log data processing method, electronic device and medium thereof |
| CN114449628B (en) * | 2021-12-30 | 2023-01-06 | 荣耀终端有限公司 | Log data processing method, electronic device and medium thereof |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110413478A (en) | A kind of method, equipment and medium monitoring log processing | |
| CN103942210A (en) | Processing method, device and system of mass log information | |
| CN108847977B (en) | Service data monitoring method, storage medium and server | |
| CN105824744B (en) | A kind of real-time logs capturing analysis method based on B2B platform | |
| WO2020233212A1 (en) | Log record processing method, server, and storage medium | |
| CN107145489B (en) | Information statistics method and device for client application based on cloud platform | |
| CN106878074B (en) | Flow filtering method and device | |
| US20180314721A1 (en) | Incremental out-of-place updates for index structures | |
| CN113360554B (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| CN104199816A (en) | Managing storage of individually accessible data units | |
| CN106407442B (en) | A kind of mass text data processing method and device | |
| CN111124839A (en) | Distributed log data monitoring method and device | |
| CN111726252A (en) | Abnormal data reporting method and tool for industrial equipment | |
| CN113391973A (en) | Internet of things cloud container log collection method and device | |
| CN105550179B (en) | Webpage collection method and browser plug-in | |
| CN106817262A (en) | A kind of log analysis device | |
| CN110019152A (en) | A kind of big data cleaning method | |
| CN105245394A (en) | Method and equipment for analyzing network access log based on layered approach | |
| JP2006221439A (en) | Method for storing log | |
| CN112506886B (en) | Multi-source service operation log acquisition method and system | |
| CN114168672B (en) | Log data processing method, device, system and medium | |
| CN105553716A (en) | Fetch and display system of each node log under large-scale cluster | |
| CN111061719B (en) | Data collection method, device, equipment and storage medium | |
| US9800684B2 (en) | Systems and methods for statistical caching | |
| CN113722416A (en) | Data cleaning method, device and equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20191105 |
|
| WW01 | Invention patent application withdrawn after publication |