CN104331487B - The processing method and processing device of log - Google Patents
The processing method and processing device of log Download PDFInfo
- Publication number
- CN104331487B CN104331487B CN201410640694.5A CN201410640694A CN104331487B CN 104331487 B CN104331487 B CN 104331487B CN 201410640694 A CN201410640694 A CN 201410640694A CN 104331487 B CN104331487 B CN 104331487B
- Authority
- CN
- China
- Prior art keywords
- log
- data acquisition
- acquisition system
- memory
- region
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1737—Details of further file system functions for reducing power consumption or coping with limited storage space, e.g. in mobile devices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Abstract
The invention discloses a kind of processing method and processing devices of log, processing method is used to handle all logs an of task, for the target journaling in all logs, processing method includes: that a region of memory is arranged in memory, parameters separated in each sentence of target journaling is come out, by parameter by a configuration information record in in the matched data acquisition system of target journaling, and the number of target journaling is recorded in data acquisition system, sentence after separation parameter is generated as log template corresponding with data acquisition system by information, record log template is numbered in data acquisition system, data acquisition system in region of memory is stored in storage disk；Generate log template number simultaneously be recorded in data acquisition system, by region of memory data acquisition system and corresponding log template be stored in storage disk.The processing method and processing device of the application can effectively save server memory consumption when record log, the time required to reducing record log and log storage.
Description
Technical field
The present invention relates to a kind of processing method and processing devices of log.
Background technique
Log is the information of hardware, software and system problem in record system, while can be to occur in monitoring system
Event.User can check the reason of mistake occurs by it, or the trace that leaves of attacker when finding under attack.Day
Will includes system log, application log and security log etc..
Because output journal needs to occupy a large amount of disk spaces, consumption disk read-write resource, system performance, Er Qie great are influenced
The log of amount also influences subsequent reading analysis, so log information output quantity must be limited.The common practice of this field with
It records similar in the patent of Publication No. CN101753344A, log content is stored in caching in the form of text, concentration is write
Enter file, occupancy caching is more, seriously affects the speed of system.
Summary of the invention
The technical problem to be solved by the present invention is to write-in, the output occupancy spatial caches in order to overcome log in the prior art
It is more, the defect of the speed of service of system is seriously affected, memory consumption when providing a kind of saving record log reduces record log
The processing method and processing device of the log of required time.
The present invention is to solve above-mentioned technical problem by following technical proposals: a kind of processing method of log, feature
It is, the processing method is used to handle all logs an of task, for the target journaling in all logs, institute
Stating processing method includes:
One region of memory is set in memory, by the ginseng in each sentence of the target journaling in the region of memory
Number is separated, by the parameter by a configuration information record in in the matched data acquisition system of the target journaling, and by institute
The number for stating target journaling is recorded in the data acquisition system, and the sentence after separation parameter is generated as and number by the configuration information
According to gathering corresponding log template,
In the case where one has the log template for storing in the storage disk of log, the log template is recorded
Be numbered in the data acquisition system, after the completion of all logs are all handled, the data acquisition system in region of memory is deposited
It is stored in storage disk；
In the case where one does not have the log template in the storage disk of log for storing, the log mould is generated
The number of plate is simultaneously recorded in the data acquisition system, after the completion of all logs are all handled, by the number in region of memory
It is stored in storage disk according to set and corresponding log template；
Wherein, the configuration information is the mapping relations of disengaged position in the parameter and sentence.
In a data processing system, needing to be implemented for a task usually has multiple logs for needing to record, Mei Ge
Include in will multiple sentences (can be description type sentence), by parameter transformable in each sentence and changeless clause point
Storage same log template can Bao Cun not be omitted on memory space, achieve the purpose that save memory space.Further, divide
Data acquisition system and log template from after are stored in disk space calling when for output, save the space of memory (caching),
The service performance of raising system.
Preferably, the treating method comprises:
After the request for receiving an output target journaling, data acquisition system is obtained according to the number of the log in request；
The parameter in data acquisition system that will acquire is written in corresponding log template according to the configuration information to revert to
Target journaling simultaneously exports.
According to the number of log, the log of the available request of system.Data acquisition system and log template in one log
It is corresponding, it include the number of corresponding log template in data acquisition system, it is available with the data set according to the numbering system
Corresponding log template is closed, the parameter in data acquisition system is written in log template the log before capable of restoring separation.
Using the number, if the log template in disk has existed, system no longer needs to send out log template
It is sent in storage disk, it is only necessary to send data acquisition system in storage disk, greatly reduce the data volume of transmission.
Preferably, the treating method comprises: data acquisition system passes through compression processing before being stored in storage disk,
Decompression processing is carried out to the data acquisition system of acquisition, by the parameter in data acquisition system after decompression according to the configuration information
It is written in corresponding log template to revert to target journaling and export.
Preferably, the memory, for multiple region of memory to be arranged, each region of memory alignment processing is different to appoint
The log of business.
The present invention also provides a kind of processing units of log, it is characterized in that, the processing unit is appointed for handling one
All logs in business, the processing unit include a setup module, a processing module and a memory module, and one is appointed
A target journaling in business,
The setup module for being arranged a region of memory in memory；
The processing module is used for the parameters separated in each sentence of the target journaling in the region of memory
Out, by the parameter by a configuration information record in in the matched data acquisition system of the target journaling, and by the target
The number of log is recorded in the data acquisition system, and the sentence after separation parameter is generated as by the configuration information and data acquisition system
Corresponding log template；
The memory module is used in the case where one has the log template for storing in the storage disk of log,
Being numbered in the data acquisition system for the log template is recorded, after the completion of all logs are all handled, by memory field
Data acquisition system in domain is stored in storage disk；
The feelings that the memory module is also used to not have the log template in a storage disk for storing log
It under condition, generates the number of the log template and is recorded in the data acquisition system, all processing is completed in all logs
Afterwards, by region of memory data acquisition system and corresponding log template be stored in storage disk；
Wherein, the configuration information is the mapping relations of disengaged position in the parameter and sentence.
Preferably, the processing unit further includes an acquisition module and a recovery module,
The module that obtains is used for after the request that reception one exports target journaling, is obtained according to the number of the log in request
Take data acquisition system；
Corresponding day is written according to the configuration information in the parameter in data acquisition system that the recovery module is used to will acquire
To revert to target journaling and export in will template.
Preferably, the processing unit further includes a compression module, the compression module in data acquisition system for being stored in
The data acquisition system is compressed before storage disk, the compression module is also used to carry out decompression processing to the data acquisition system of acquisition, so
Call the recovery module that corresponding log template is written according to the configuration information in the parameter in data acquisition system after decompression afterwards
In to revert to target journaling and export.
Preferably, the multiple region of memory of memory setting, the different task of each region of memory alignment processing
Log.
On the basis of common knowledge of the art, above-mentioned each optimum condition, can any combination to get each preferable reality of the present invention
Example.
The positive effect of the present invention is that: when the processing method and processing device of the application can effectively save record log
Server memory consumption, reduce record log and log storage the time required to.Volume of transmitted data greatly reduces, due to service
Device does not need to handle complete log, it is only necessary to parameter needed for handling log template serial number and template, therefore lacked memory and make
With improving the performance of system.
Detailed description of the invention
Fig. 1 is the flow chart of the processing method of the embodiment of the present invention.
Fig. 2 is the structural schematic diagram of the processing unit of the embodiment of the present invention.
Specific embodiment
The present invention is further illustrated below by the mode of embodiment, but does not therefore limit the present invention to the reality
It applies among a range.
Embodiment
Referring to fig. 2, the present embodiment provides a kind of processing units 1 of log, and the processing unit of the present embodiment is for handling two
Log in a task, wherein include 3 logs in each task, the processing unit includes at a setup module 11, one
Manage module 12, a memory module 13, one obtains module 14, a recovery module 15 and a compression module 16.
For the target journaling in a task:
The setup module for being arranged a region of memory in memory, and 3 logs in the task are in memory field
It is handled in domain, the region of memory only handles the log of a task, another task is handled in another region of memory.
The processing module is used for the parameters separated in each sentence of the target journaling in the region of memory
Out, by the parameter by a configuration information record in in the matched data acquisition system of the target journaling, and by the target
The number of log is recorded in the data acquisition system, and the sentence after separation parameter is generated as by the configuration information and data acquisition system
Corresponding log template.Wherein, the configuration information is the mapping relations of disengaged position in the parameter and sentence.
For recording the log in the case where one has the log template for storing in the storage disk of log
Template is numbered in the data acquisition system, after the completion of all logs are all handled, by the data set in region of memory
Conjunction is stored in storage disk.
The feelings that the memory module is also used to not have the log template in a storage disk for storing log
It under condition, generates the number of the log template and is recorded in the data acquisition system, all processing is completed in all logs
Afterwards, by region of memory data acquisition system and corresponding log template be stored in storage disk.
The module that obtains is used for after the request that reception one exports target journaling, is obtained according to the number of the log in request
Take data acquisition system.
The compression module before data acquisition system is stored in storage disk for compressing the data acquisition system, the compression mould
Block is also used to carry out decompression processing to the data acquisition system of acquisition, then calls the recovery module will be in data acquisition system after decompression
Parameter is written in corresponding log template according to the configuration information to revert to target journaling and export.
The multiple region of memory of memory setting, the log of the different task of each region of memory alignment processing.
Referring to Fig. 1, using above-mentioned processing unit, the present embodiment also provides a kind of processing method of log, the processing side
The log of method processing is equipped with number, 3 logs in a task is handled using the processing method, for one of those
The target journaling that number is 19, the target journaling are as follows:
1024 bytes are had sent to No. 5 servers；
2048 bytes are had sent to No. 7 servers.
For above-mentioned target journaling, the treating method comprises:
A region of memory is arranged in step 100 in memory, by each language of the target journaling in the region of memory
Sentence in parameters separated come out, by the parameter by a configuration information record in the matched data acquisition system of the target journaling
It is interior, and the number of the target journaling is recorded in the data acquisition system, the sentence after separation parameter presses the configuration information
It is generated as log template corresponding with data acquisition system, wherein the configuration information is disengaged position in the parameter and sentence
Mapping relations.
That is, 5,1024,7,2048 be to need parameter to be separated in two sentences in the target journaling,
By 5,1024,7,2048 according to the configuration information record in the log mould that in data acquisition system, the sentence after separation parameter is generated
Plate are as follows:
{ 1 } a byte is had sent to { 0 } number server；
{ 1 } a byte is had sent to { 0 } number server.
Data acquisition system are as follows:
19,1377；
5,1024；
7,2048；
Above-mentioned data acquisition system is generated according to configuration information, so that the first row of data acquisition system indicates the identity of target journaling
And corresponding log template, the second row indicate specific parameter, the line number of parameter is corresponding with log template, that is, utilizes configuration
Information, the parameter can be corresponding with disengaged position in sentence.
Also there is the number 1377 of this log template in the data acquisition system, restores the target journaling for system.
There is journal number 19 in the data acquisition system and calls the target journaling for system.
Step 101 judges whether there is the log template in a storage disk for storing log, if then holding
Row step 102, thens follow the steps 103 if not.
Step 102 records being numbered in the data acquisition system of the log template, all handles in all logs
After the completion, the data acquisition system in region of memory is stored in storage disk, then executes step 104.
Step 103, the number for generating the log template are simultaneously recorded in the data acquisition system, complete in all logs
Portion processing after the completion of, by region of memory data acquisition system and corresponding log template be stored in storage disk.
Wherein, data acquisition system passes through compression processing before being stored in storage disk, so that log occupies the space of disk more
It is small.In the storage disk, if the log template that number is 1377 has existed, system is judging that log template deposited
Only the corresponding data acquisition system of log template of number 1377 can be stored in storage disk in case, and no longer stored
Log template, to reduce the transmission quantity of data and the space hold amount of storage disk.
Step 104 after receiving the request that one exports target journaling, obtains data set according to the number of the log in request
It closes.
Step 105 carries out decompression processing to the data acquisition system of acquisition, by the parameter in data acquisition system after decompression according to described
Configuration information is written in corresponding log template to revert to target journaling and export.
At step 104, the log that output number is 19 is required in the request, system obtains data set according to number 19
Close, and the corresponding log template of data acquisition system obtained according to number 1377, then in a caching by data acquisition system 5,
1024, it 7,2048 is filled into log template, restores target journaling, and the target journaling is exported.
The processing method and processing device of the present embodiment can effectively save server memory consumption when record log, reduce note
The time required to recording log and log storage.Volume of transmitted data greatly reduces, since server does not need to handle complete log,
Parameter needed for only needing to handle log template serial number and template, therefore lacked memory use, improve the performance of system.
Although specific embodiments of the present invention have been described above, it will be appreciated by those of skill in the art that these
It is merely illustrative of, protection scope of the present invention is defined by the appended claims.Those skilled in the art is not carrying on the back
Under the premise of from the principle and substance of the present invention, many changes and modifications may be made, but these are changed
Protection scope of the present invention is each fallen with modification.
Claims (6)
1. a kind of processing method of log, which is characterized in that the processing method is used to handle all logs an of task, right
A target journaling in all logs, the treating method comprises:
One region of memory is set in memory, by the parameter in each sentence of the target journaling point in the region of memory
Separate out and, by the parameter by a configuration information record in in the matched data acquisition system of the target journaling, and by the mesh
The number of mark log is recorded in the data acquisition system, and the sentence after separation parameter is generated as by the configuration information and data set
Corresponding log template is closed,
In the case where one has the log template for storing in the storage disk of log, the volume of the log template is recorded
Number in the data acquisition system, after the completion of all logs are all handled, the data acquisition system in region of memory is stored in
In storage disk；
In the case where one does not have the log template in the storage disk of log for storing, the log template is generated
It numbers and is recorded in the data acquisition system, after the completion of all logs are all handled, by the data set in region of memory
It closes and corresponding log template is stored in storage disk；
Wherein, the configuration information is the mapping relations of disengaged position in the parameter and sentence；It the treating method comprises:
After the request for receiving an output target journaling, data acquisition system is obtained according to the number of the log in request；
The parameter in data acquisition system that will acquire is written in corresponding log template according to the configuration information to revert to target
Log simultaneously exports.
2. processing method as described in claim 1, which is characterized in that the treating method comprises: data acquisition system is being stored in
Pass through compression processing before storage disk,
Decompression processing is carried out to the data acquisition system of acquisition, the parameter in data acquisition system after decompression is written according to the configuration information
To revert to target journaling and export in corresponding log template.
3. processing method as described in claim 1, which is characterized in that the memory is used to that multiple region of memory to be arranged,
The log of the different task of each region of memory alignment processing.
4. a kind of processing unit of log, which is characterized in that the processing unit is used to handle all logs in a task,
The processing unit includes a setup module, a processing module and a memory module, for the target day in a task
Will,
The setup module for being arranged a region of memory in memory；
The processing module is used to come out the parameters separated in each sentence of the target journaling in the region of memory,
By the parameter by a configuration information record in in the matched data acquisition system of the target journaling, and by the target journaling
Number is recorded in the data acquisition system, and the sentence after separation parameter is generated as corresponding with data acquisition system by the configuration information
Log template；
The memory module is used in the case where one has the log template for storing in the storage disk of log, record
The log template is numbered in the data acquisition system, will be in region of memory after the completion of all logs are all handled
Data acquisition system be stored in storage disk；
The memory module is also used in the case where one does not have the log template in the storage disk of log for storing,
It generates the number of the log template and is recorded in the data acquisition system, it, will after the completion of all logs are all handled
Data acquisition system and corresponding log template in region of memory are stored in storage disk；
Wherein, the configuration information is the mapping relations of disengaged position in the parameter and sentence；The processing unit further includes
One obtains module and a recovery module,
The module that obtains is used for after the request that reception one exports target journaling, obtains number according to the number of the log in request
According to set；
Corresponding log mould is written according to the configuration information in the parameter in data acquisition system that the recovery module is used to will acquire
To revert to target journaling and export in plate.
5. processing unit as claimed in claim 4, which is characterized in that the processing unit further includes a compression module, described
For compressing the data acquisition system before data acquisition system is stored in storage disk, the compression module is also used to obtaining compression module
The data acquisition system taken carries out decompression processing, then calls the recovery module by the parameter in data acquisition system after decompression according to described
Configuration information is written in corresponding log template to revert to target journaling and export.
6. processing unit as claimed in claim 4, which is characterized in that the multiple region of memory of memory setting, it is each
The log of the different task of region of memory alignment processing.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410640694.5A CN104331487B (en) | 2014-11-13 | 2014-11-13 | The processing method and processing device of log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410640694.5A CN104331487B (en) | 2014-11-13 | 2014-11-13 | The processing method and processing device of log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104331487A CN104331487A (en) | 2015-02-04 |
| CN104331487B true CN104331487B (en) | 2019-03-12 |
Family
ID=52406214
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410640694.5A Active CN104331487B (en) | 2014-11-13 | 2014-11-13 | The processing method and processing device of log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104331487B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106326082B (en) * | 2015-06-16 | 2020-10-16 | 南京中兴新软件有限责任公司 | Method and device for recording log in network system |
| CN105975376A (en) * | 2016-04-28 | 2016-09-28 | 广州市锲致智能技术有限公司 | Log processing based numerical control system security processing method |
| WO2019161551A1 (en) * | 2018-02-24 | 2019-08-29 | 黄裕才 | Method for processing log based on logrotate technology |
| CN108874639A (en) * | 2018-03-27 | 2018-11-23 | 平安科技（深圳）有限公司 | Log archiving method, electronic device and readable storage medium storing program for executing |
| CN109684290B (en) * | 2018-12-20 | 2021-08-17 | 东软集团股份有限公司 | Log storage method, device, equipment and computer readable storage medium |
| CN109885545A (en) * | 2019-02-02 | 2019-06-14 | 华为技术有限公司 | It stores, the method, apparatus of inquiry log information |
| CN111538712B (en) * | 2020-04-30 | 2023-07-21 | 恒生电子股份有限公司 | Log recording method, processing node, electronic equipment and storage medium |
| CN112286896B (en) * | 2020-11-19 | 2024-02-06 | 北京百家科技集团有限公司 | Log file processing method and device, computer equipment and storage medium |
| CN112559458B (en) * | 2020-12-11 | 2021-11-30 | 深圳金信诺高新技术股份有限公司 | Method and system for reducing log size |
| CN113179265A (en) * | 2021-04-26 | 2021-07-27 | 湖南蚁坊软件股份有限公司 | Log compression encryption method, device, equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8321433B1 (en) * | 2011-05-06 | 2012-11-27 | Sap Ag | Systems and methods for business process logging |
| CN103412924B (en) * | 2013-08-12 | 2017-10-31 | 东软集团股份有限公司 | log multi-language query method and system |
| CN103440297A (en) * | 2013-08-20 | 2013-12-11 | 苏州迈科网络安全技术股份有限公司 | Method for recording and reading user operation log information |
| CN103577307A (en) * | 2013-11-07 | 2014-02-12 | 浙江中烟工业有限责任公司 | Method for automatically extracting and analyzing firewall logs based on XML rule model |
-
2014
- 2014-11-13 CN CN201410640694.5A patent/CN104331487B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN104331487A (en) | 2015-02-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104331487B (en) | The processing method and processing device of log | |
| CN104104717B (en) | Deliver channel data statistical approach and device | |
| US9037555B2 (en) | Asynchronous collection and correlation of trace and communications event data | |
| CN104317709B (en) | software performance testing method and system | |
| CN104408190B (en) | Data processing method and device based on Spark | |
| US20150350045A1 (en) | Method and system for processing browser crash information | |
| CN104572415B (en) | A kind of transaction log record method suitable for distributed system | |
| CN104022913B (en) | For method of testing and the device of data cluster | |
| CN106534242A (en) | Processing method and device for requests in distributed system | |
| CN108197008A (en) | A kind of log collecting method, system, device and computer readable storage medium | |
| CN103580899B (en) | Event log management method, system, cloud service client and virtual platform | |
| CN106569750A (en) | Data compression method and device | |
| CN103812679B (en) | A kind of massive logs statistical analysis system and method | |
| CN107872713A (en) | Short processing system for video, method and device | |
| CN104298918B (en) | A kind of virus scan method and system in virtual machine based on data block | |
| CN206021244U (en) | A kind of data collecting system under distributed computer cluster | |
| CN103236938A (en) | Method and system for user action collection based on cache memory and asynchronous processing technology | |
| CN106610894A (en) | Interface testing method and device | |
| CN109522273A (en) | A kind of method and device for realizing data write-in | |
| CN108696713A (en) | Safety detecting method, device and the test equipment of code stream | |
| CN107239572A (en) | The data cache method and device of a kind of storage management software | |
| CN107145414B (en) | Method and system for testing distributed object storage | |
| CN106254806A (en) | A kind of Video data guard method and device | |
| CN108874645A (en) | A kind of method and system for integrating FIO performance data | |
| CN109656945A (en) | A kind of information processing method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20160204 Address after: 200335 Shanghai city Changning District Admiralty Road No. 968 Building No. 16 10 floor Applicant after: SHANGHAI XIECHENG BUSINESS CO., LTD. Address before: 200335 Shanghai City, Changning District Fuquan Road No. 99, Ctrip network technology building Applicant before: Ctrip computer technology (Shanghai) Co., Ltd. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |