CN110515898A - A kind of log processing method and device - Google Patents
A kind of log processing method and device Download PDFInfo
- Publication number
- CN110515898A CN110515898A CN201910704328.4A CN201910704328A CN110515898A CN 110515898 A CN110515898 A CN 110515898A CN 201910704328 A CN201910704328 A CN 201910704328A CN 110515898 A CN110515898 A CN 110515898A
- Authority
- CN
- China
- Prior art keywords
- log
- file
- index
- journal file
- journal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The present invention provides a kind of log processing method and device, this method comprises: the response time needed for indexing journal file according to log index file, or, it is cut according to the time of the occurrence log recorded in log index file, determine the log index file to meet certain requirements, the log index file for meeting certain requirements and be pre-stored within index database is stored to distributed file system, and relevant processing operation is executed to the journal file that the log index file indexes by preset rules.In embodiments of the present invention, the log index file to meet certain requirements in index database is migrated to distributed file system, and journal file is indexed to log index file and is managed.It can be improved the operational efficiency of journal file and the space utilization rate of index database.
Description
Technical field
The present invention relates to technical field of information processing more particularly to a kind of log processing method and device.
Background technique
With the development of science and technology, the development of computer technology is also getting faster, in order to understand and safeguard department of computer science
The operation conditions of system and application is right often by log recording computer system health information and application processing data information
The journal file of record is indexed, and the file of index is stored in local spatial.
In big data platform, computer system and application to handle data quantity it is huger, log needs to record
Computer system health information and application processing data information quantity increase therewith.On the one hand due in index day
Not to data management during will file.Cause the operational efficiency of log low.On the other hand due to the local spatial of log
Capacity be limited, whole data can not be stored, easily cause loss of data, so as to cause the space utilization rate of log
It is low.
Therefore, the operational efficiency and space utilization rate for how improving log are urgent problems to be solved of the present invention.
Summary of the invention
In view of this, the embodiment of the present invention provides a kind of log processing method and device, to solve log in the prior art
Operational efficiency is low and and the lower problem of space utilization rate.
To achieve the above object, the embodiment of the present invention provides the following technical solutions:
First aspect present invention discloses a kind of log processing method, which comprises
Response time needed for indexing journal file according to log index file, alternatively, remembering according in log index file
The time of the occurrence log of record cuts, and the log index file for being pre-stored within index database is stored to distributed file system；
Relevant processing operation is executed to the journal file that the log index file indexes according to preset rules.
It is optionally, described in advance in the process of index database storage log index file, comprising:
Collector journal, and journal file is established according to Log Types；
According to the essential information of log each in journal file, log index file is established, wherein in the journal file
The essential information of each log includes the time section of logger computer system operation situation information or application processing data information, sequence
Number and log size；
The log index file is stored to index database.
Optionally, the response time needed for the index journal file according to log index file, alternatively, according to log rope
The time of the occurrence log recorded in quotation part cuts, and the log index file for being pre-stored within index database is stored to distributed text
Part system includes:
According to the log index file, the essential information of log is obtained, the essential information of the log includes recording gauge
The time of calculation machine system operation situation information or application processing data information cuts；
Each journal file is traversed, journal file corresponding with the essential information of the log is indexed；
Response time needed for the corresponding journal file of the essential information of recording indexes to the log；
Whether the response time needed for indexing the corresponding journal file of essential information of log described in judgement is more than or equal to
The preset response time；
The response time needed for the corresponding journal file of the essential information for indexing log is more than or equal to preset
It is according to the store path and title for presetting the distributed file system, the journal file is corresponding when the response time
Log index file migrate to the distributed file system.
Optionally, the response time needed for the index journal file according to log index file, alternatively, according to log rope
The time of the occurrence log recorded in quotation part cuts, and the log index file for being pre-stored within index database is stored to distributed text
Part system includes:
According to the log index file, the essential information of log is obtained, the essential information of the log includes recording gauge
The time of calculation machine system operation situation information or application processing data information cuts；
Each journal file is traversed, journal file corresponding with the essential information of the log is indexed；
Response time needed for the corresponding journal file of the essential information of recording indexes to the log；
Whether the time of the computer system health information or application processing data information that judge the record cuts small
It is cut in the preset time；
When the computer system health information of the record or the time of application processing data information cut to be less than and preset
Time when cutting, it is according to the store path and title for presetting the distributed file system, the journal file is corresponding
Log index file migrate to the distributed file system.
Optionally, described that relevant place is executed to the journal file that the log index file indexes according to preset rules
Reason operation, comprising:
External interface is called, data parameters are generated；
Match operating parameter corresponding with the data parameters in preset database；
Filing is executed according to the journal file that the operating parameter indexes the log index file, removes or saves
Operation.
Optionally, the method also includes:
It is grasped when the journal file that the log index file indexes is executed filing or saved according to the operating parameter
When making, the journal file is filed or is saved to the storage system of predesignated storage location.
Optionally, the method also includes:
According to preset Hyper text transfer security protocol, the journal file and the log index file are shown.
Second aspect of the present invention discloses a kind of log processing device, and described device includes: log management unit and log
Indexing units；
The log indexing units, for the response time needed for indexing journal file according to log index file, alternatively,
It is cut according to the time of the occurrence log recorded in log index file, the log index file for being pre-stored within index database is stored
To distributed file system；
The log management unit, the journal file for being indexed according to preset rules to the log index file are held
The relevant processing operation of row.
Optionally, the log management unit is provided with external interface, and the log management unit includes:
Generation unit generates data parameters for calling external interface；
Matching unit, for matching operating parameter corresponding with the data parameters in preset database；
Execution unit, for being returned according to the operating parameter to the journal file execution that the log index file indexes
Operation is removed or saved to shelves.
Optionally, described device further include: blog search boundary element；
The blog search boundary element, for showing the log text according to preset Hyper text transfer security protocol
Part and the log index file.
Based on log processing method and device that the embodiments of the present invention provide, this method comprises: being indexed according to log
Response time needed for file index journal file, alternatively, cut according to the time of the occurrence log recorded in log index file,
It determines the log index file to meet certain requirements, will meet certain requirements and be pre-stored within the log index file of index database
It stores to distributed file system, and correlation is executed by the journal file that preset rules index the log index file
Processing operation.In embodiments of the present invention, the index journal file recorded when indexing journal file according to log index file
The time of the occurrence log recorded in the required response time and log index file cuts, so that it is determined that meeting the log of condition
Index file migrates the log index file to meet certain requirements in index database to distributed file system, and to log rope
Quotation part indexes journal file and is managed.To improve the operational efficiency of journal file and the space utilization rate of index database.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of invention for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of flow diagram of log processing method provided in an embodiment of the present invention；
Fig. 2 is the flow diagram of another log processing method provided in an embodiment of the present invention；
Fig. 3 specifically stores log index file to the flow diagram of HDFS to be provided in an embodiment of the present invention；
Fig. 4 is provided in an embodiment of the present invention another specifically stores log index file to the flow diagram of HDFS；
Fig. 5 is the specific flow diagram that relevant processing operation is executed to journal file that present invention implementation provides；
Fig. 6 is a kind of structural schematic diagram of log processing device provided in an embodiment of the present invention；
Fig. 7 is the structural schematic diagram of another log processing device provided in an embodiment of the present invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
In this application, the terms "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion,
So that the process, method, article or equipment for including a series of elements not only includes those elements, but also including not having
The other element being expressly recited, or further include for elements inherent to such a process, method, article, or device.Do not having
There is the element limited in the case where more limiting by sentence "including a ...", it is not excluded that in the mistake including the element
There is also other identical elements in journey, method, article or equipment.
By above-mentioned background technique it is found that in big data platform, computer system and application will handle the quantity ratios of data
It is huger, the increasing therewith of the quantity of computer system health information and application processing data information that log needs to record
Add.Therefore, on the one hand due to during indexing journal file not to data management.Cause the operational efficiency of log low.
On the other hand since the capacity of the local spatial of log is limited, whole data can not be stored, loss of data is easily caused,
It is low so as to cause the space utilization rate of log.
Therefore, the embodiment of the present invention provides a kind of log processing method and device, indexes log according to log index file
The process of file, is managed journal file, and by the log index file to meet certain requirements in index database migrate to point
Cloth file system, to improve the operational efficiency of journal file and the space utilization rate of index database.
Be a kind of flow diagram of log processing method provided in an embodiment of the present invention referring to Fig. 1, this method include with
Lower step:
Step S101: the response time needed for indexing journal file according to log index file, alternatively, being indexed according to log
The time of the occurrence log recorded in file cuts, and the log index file for being pre-stored within index database is stored to distributed document
System (Hadoop Distributed File System, HDFS).
In step s101, the log index file includes: the time of the content of each log, each log generation
It cuts, the type of the serial number of each log, the corresponding log size of each log-sequence numbers and each log etc. is for characterizing log text
This can be set according to actual conditions in the essential information of part attribute, and the application is without restriction.
It should be noted that the serial number of the log includes the number of putting in order and offset, the offset referred to day
The distance between the sector address of actual address and the log that will file is stored.
During implementing step S101, according to when response needed for log index file index journal file
Between, alternatively, according to the computer system health information that is recorded in log index file or application processing data information when
Between cut, whether meet some requirements to determine that the response time and the time cut, by what is met certain condition, and in advance
The log index file for being stored in index database is migrated to HDFS.
It should be noted that described in advance in the process of index database storage log index file, comprising:
Collector journal, and journal file is established according to the type of log.
Wherein, the type of the log includes type of error, Exception Type or starting type etc., can be set according to the actual situation
Fixed, the application is without restriction.
According to the essential information of log each in journal file, log index file is established.
Wherein, in the journal file essential information of each log include logger computer system operation situation information or
Using the size of the time section of processing data information, serial number and log.
The log index file is stored to index database.
It should be noted that the index database is local data store.
Step S102: relevant processing is executed to the journal file that the log index file indexes according to preset rules
Operation.
During implementing step S102, the log index file is indexed according to preparatory rule log
File executes removing, preservation or archive operation.
It should be noted that the journal file includes the type of log, computer system health information or application
Data information is handled, the time of log cuts, the offset etc. of log.
In embodiments of the present invention, the response time needed for indexing journal file by log index file, alternatively, log
The time of the occurrence log recorded in index file cuts, and determines the log index file to meet certain requirements, satisfaction is had to
It asks and the log index file for being pre-stored within index database is stored to distributed file system, and by preset rules to the day
The journal file that will index file indexes executes relevant processing operation.In embodiments of the present invention, text is indexed according to log
The generation day recorded in response time needed for part indexes the index journal file recorded when journal file and log index file
The time of will cuts, so that it is determined that meeting the log index file of condition, the log to meet certain requirements in index database is indexed text
Part is migrated to distributed file system, and is indexed journal file to log index file and be managed.It can be improved log text
The operational efficiency of part and the space utilization rate of index database.
Based on embodiments of the present invention log processing method shown in fig. 1, step S102 is being executed according to preset rules
After the specific implementation process for executing relevant processing operation to the journal file that the log index file indexes, in conjunction with figure
1, with reference to Fig. 2, the log processing method further include:
Step S103: according to preset Hyper text transfer security protocol, the journal file and log index are shown
The content of file.
During implementing step S103, according to preset Hyper text transfer security protocol, by the log rope
Quotation part and the journal file are encrypted, and need to browse in the journal file and the log index file in user
Rong Shi by being decrypted according to preset Hyper text transfer security protocol, and shows the journal file and the log rope
The content of quotation part.
It should be noted that the process for presetting Hyper text transfer security protocol includes:
Key keystore file and key truststore file are generated according to key and certificate tool, in configuration file
Middle introducing key keystore file and key truststore file, thus by user interface (User Interface, UI)
Protocol is Hyper text transfer security protocol.
It should be noted that key keystore file saves the private key certificate generated, key truststore file is protected
Deposit the public key certificate of generation.
In embodiments of the present invention, by setting Hyper text transfer security protocol, allow user clear in a browser
The content of journal file and log index file is look at, to improve the safety at log index interface.
Based on embodiments of the present invention log processing method shown in fig. 1, indexed executing step S101 according to log
Response time needed for file index journal file, alternatively, cut according to the time of the occurrence log recorded in log index file,
During the log index file for being pre-stored within index database is stored to distributed file system, as shown in figure 3, include with
Lower step:
Step S301: according to the log index file, the essential information of log is obtained.
In step S301, the essential information of the log includes at least logger computer system operation situation information or answers
Time section and serial number with processing data information.
Step S302: traversing each journal file, indexes journal file corresponding with the essential information of the log.
During implementing step S302, each journal file is searched by sequential search method, search with it is described
The corresponding journal file of serial number in the essential information of log.
Step S303: response time needed for the corresponding journal file of the essential information of recording indexes to the log.
Step S304: whether the response time needed for indexing the corresponding journal file of essential information of log described in judgement
More than or equal to the preset response time, the response time needed for the corresponding journal file of the essential information for indexing log
When more than or equal to the preset response time, step S305 is executed, conversely, judging needed for next journal file indexed
Whether the response time is more than or equal to the preset response time.
It should be noted that the preset response time is the test value of preparatory test of many times.
Step S305: according to the store path and title for presetting the distributed file system, by the log text
The corresponding log index file of part is migrated to the distributed file system.
In order to better understand the content shown in above-mentioned Fig. 3, it is exemplified below.
Assuming that the path of distributed file system is computer D disk, the entitled F of distributed file system, preset sound
It is 8ms between seasonable.According to the log index file, the serial number 1,3480 of log a is obtained, each journal file is traversed, indexed
Journal file A corresponding with the essential information of the log, the serial number 1 of recording indexes to log a, 3480 corresponding journal files
Response time needed for A is 10ms, determines the response time needed for indexing the corresponding journal file A of essential information of log a
10ms is more than or equal to preset response time 8ms, and the corresponding log index file of journal file A is migrated to computer D disk,
In the cloth file system of entitled F.
In the embodiment of the present invention, the response time needed for the corresponding journal file of essential information by indexing log,
The length for determining the response time migrates the corresponding log index file of journal file to the distributed file system.It can
Local database space is reduced in time, to promote local spatial utilization rate.
Based on the response time according to needed for log index file index journal file shown in above-mentioned Fig. 3, alternatively, according to
The time of the occurrence log recorded in log index file cut, by the log index file for being pre-stored within index database store to point
The process of cloth file system, as shown in figure 4, text can also be indexed for the log for being pre-stored within index database in the following manner
Part is stored to distributed file system, specific implementation procedure the following steps are included:
Step S401: according to the log index file, the essential information of log is obtained.
In step S401, the essential information of the log includes at logger computer system operation situation information or application
The time for managing data information cuts.
Step S402: traversing each journal file, indexes journal file corresponding with the essential information of the log.
It should be noted that in addition to corresponding journal file and day can be searched by the serial number in the essential information of log
The time of computer system operation situation information or application processing data information cuts to search corresponding day in the essential information of will
Will file can also search corresponding journal file by data other in log essential information, this application is not added
With limitation.
Step S403: response time needed for the corresponding journal file of the essential information of recording indexes to the log.
Step S404: judge the computer system health information of the record or the time of application processing data information
It cuts and whether is less than the preset time section.If the computer system health information of the record or application handle data information
Time, which cuts, is less than the preset time section, thens follow the steps S405, conversely, then judging needed for next journal file indexed
Response time whether be more than or equal to the preset response time.
During implementing step S404, by judge the record computer system health information or
It is cut using the time of processing data information and whether is less than the preset time section, so that it is determined that the computer system of the record is run
Query time, is less than the day of preset time section by the query time of the journal file of condition information or application processing data information
Will file is migrated.The preset time, which cuts, may be configured as the previous year of current time, can according to the actual situation depending on, this
Apply without restriction.
Step S405: according to the store path and title for presetting the distributed file system, by the log text
The corresponding log index file of part is migrated to the distributed file system.
It should be noted that including multiple log file datas for being distributed in different nodes in the log index file
Index fragment.
It should be noted that response needed for showing settled date will index file index journal file really in addition to above-mentioned Fig. 3
Mode and above-mentioned Fig. 4 that whether the time meets certain condition show the occurrence log recorded in settled date will index file really
Time cuts outside the mode that whether meets certain condition goes to migrate the log index file for being pre-stored within index database, may be used also
To determine whether journal file is the journal file of alarm type according to the type of journal file, so that it is determined that not being alarm class
The corresponding log index file of the journal file of type migrates the log index file for being pre-stored within index database.
Further, it should be noted that the alarm log file is the journal file of real time monitoring.
In order to better understand the content shown in above-mentioned Fig. 4, it is exemplified below.
Assuming that the path of distributed file system is computer D disk, the entitled F of distributed file system, journal file A
20 divide 36 seconds when the time of the database application processing log-on data information of middle record is truncated into 1 day 10 July in 2013；Assuming that default
Time when being truncated into 1 day 0 January in 2014；Determine recorded in journal file A database application processing log-on data information when
Between cut 1 day 10 July in 2013 when 20 divide 36 seconds be less than the preset time cut 1 day 0 January in 2014, journal file A is corresponding
Log index file is migrated to computer D disk, in the cloth file system of entitled F.
In the embodiment of the present invention, pass through the computer system health information or application processing data information of log recording
Time cut, so that it is determined that the size that the time cuts, the corresponding log index file of journal file is migrated to the distributed text
Part system.Local space can be reduced in time, to promote local spatial utilization rate.
Based on embodiments of the present invention log processing method shown in fig. 1, step 102 is being executed according to preset rules pair
The journal file that the log index file indexes executes relevant processing operation, comprising the following steps:
Step S501: calling external interface, generates data parameters.
It should be noted that the data parameters include: data manipulation mode, data source address, filtered fields, read-write behaviour
Make and size etc., to this can according to the actual situation depending on, the application is without restriction.
Step S502: operating parameter corresponding with the data parameters in preset database is matched.
During implementing step S502, operating parameter corresponding with data parameters is set in database in advance,
For example data parameters are data manipulation mode m ade, corresponding operating parameter is delete, and the operating parameter delete is used for
Characterization carries out delete operation to journal file.
Step S503: according to the journal file that the operating parameter indexes the log index file execute filing,
Remove or save operation.
During implementing step S503, if the operating parameter is purged behaviour to journal file for characterizing
Make, then removes journal file from local storage system.
It should be noted that the journal file is stored in local storage system, HDFS etc., can according to the actual situation and
Fixed, the application is without restriction.
Such as, it is assumed that external interface is called, data parameters data manipulation mode m ode is generated, matches preset data
Operating parameter archive corresponding with the data parameters in library, the operating parameter archive are for characterizing to journal file
Carry out archive operation.
In the bright embodiment of we, by determining the corresponding operating parameter of data parameters, and operation is executed to journal file
The corresponding operation of parameter, can be realized the management to journal file, to improve the operational efficiency and index database of journal file
Space utilization rate.
Based on embodiments of the present invention log processing method shown in Fig. 4, execute the step S503 shown in Fig. 5 according to
After the journal file that the operating parameter indexes the log index file executes filing, removes or saves operation, also
Include:
It is grasped when the journal file that the log index file indexes is executed filing or saved according to the operating parameter
When making, the journal file is filed or is saved to the storage system of predesignated storage location.
In implementation, the operation that the operating parameter is characterized is determined, when the operation characterized is archive operation or preservation behaviour
When making, the journal file is filed or is saved to the storage system of predesignated storage location.Such as: assuming that determining the behaviour
Make the operation that parameter is characterized to save operation, designated storage location is computer D disk, and storage system is distributed file system
A saves the journal file into the distributed file system of computer D disk.
It should be noted that the storage system include local data base and HDFS etc. support the storage of a variety of files be
System, can be set according to actual conditions, the application is without restriction.
In embodiments of the present invention, when the operation for determining that the operating parameter is characterized is archive operation or saves operation,
Journal file is filed or is saved to the storage system of predesignated storage location, the position of journal file can be specified, so that
Preferably journal file is managed.
Corresponding with log processing method disclosed in the embodiments of the present invention, also correspondence of the embodiment of the present invention discloses one
Kind log processing device, as shown in fig. 6, the log processing device includes: log management unit 601 and log indexing units
602。
The log indexing units 601, for the response time needed for indexing journal file according to log index file, or
Person cuts according to the time of the occurrence log recorded in log index file, will be pre-stored within the log index file of index database
It stores to distributed file system；
The log management unit 602, the log text for being indexed according to preset rules to the log index file
Part executes relevant processing operation.
It should be noted that the specific principle of each unit in log processing device disclosed in the embodiments of the present invention
And implementation procedure, it is identical as aforementioned present invention implementation log processing method, reference can be made to log disclosed in the embodiments of the present invention
Corresponding part, is not discussed here in processing method.
In embodiments of the present invention, by log indexing units, needed for indexing journal file according to log index file
Response time determines the log to meet certain requirements alternatively, cutting according to the time of the occurrence log recorded in log index file
Index file stores the log index file for meeting certain requirements and be pre-stored within index database to distributed file system,
And by log management unit, relevant place is executed to the journal file that the log index file indexes according to preset rules
Reason operation.In embodiments of the present invention, needed for the index journal file recorded when indexing journal file according to log index file
Response time and log index file in time of occurrence log for recording cut, so that it is determined that meeting the log index of condition
File migrates the log index file to meet certain requirements in index database to distributed file system, and indexes text to log
Part indexes journal file and is managed.To improve the operational efficiency of journal file and the space utilization rate of index database.
Based on the log processing device shown in above-mentioned Fig. 6, offer of the embodiment of the present invention is shown with reference to Fig. 7 in conjunction with Fig. 6
A kind of log processing device structural block diagram, described device further include: blog search boundary element 603.
The blog search UI unit 603, for showing the log text according to preset Hyper text transfer security protocol
Part and the log index file.
In the embodiment of the present invention, indexing UI unit introducing Hyper text transfer security protocol by log can be clear in browser
The content of journal file and log index file is look at, to improve the safety of log index UI.
Based on the log processing device shown in above-mentioned Fig. 6, the log management unit 601 is provided with external interface, described
Log management unit 601 includes:
Generation unit generates data parameters for calling external interface.
Matching unit, for matching operating parameter corresponding with the data parameters in preset database.
Execution unit, for being returned according to the operating parameter to the journal file execution that the log index file indexes
Operation is removed or saved to shelves.
In embodiments of the present invention, the corresponding operation of operating parameter, energy are executed to journal file by log management unit
Enough management realized to journal file, to improve the operational efficiency of journal file and the space utilization rate of index database.
Further, based on above-mentioned Fig. 6 shown in log processing device, the log management unit 601 further include storage
Unit, the storage unit are arranged on execution unit.
The storage unit, for when the journal file indexed according to the operating parameter to the log index file
When executing filing or saving operation, the journal file is filed or is saved to the storage system of predesignated storage location.
In the embodiment of the present invention, journal file is filed to by storage unit or is saved depositing to predesignated storage location
Storage system can specify the position of journal file, log management unit is preferably managed journal file.
Based on the log processing device shown in above-mentioned Fig. 6, the log indexing units include:
First acquisition unit, for obtaining the essential information of log, the base of the log according to the log index file
This information includes the time section of logger computer system operation situation information or application processing data information.
First indexing units index log corresponding with the essential information of the log for traversing each journal file
File.
First recording unit, response needed for the corresponding journal file of essential information for recording indexes to the log
Time.
First judging unit, for judge it is described index the corresponding journal file of essential information of log needed for response
Whether the time is more than or equal to the preset response time.
First migration units are big for the response time needed for indexing the corresponding journal file of essential information of log
It, will be described according to the store path and title for presetting the distributed file system when being equal to the preset response time
The corresponding log index file of journal file is migrated to the distributed file system.
In embodiments of the present invention, pass through the corresponding journal file of essential information for indexing log of recording unit records
The required response time, so that it is determined that the length of response time, the corresponding log index file of journal file is migrated to described
Distributed file system.Local space can be reduced in time, to promote local spatial utilization rate.
Based on the log processing device shown in above-mentioned Fig. 6, the log indexing units further include:
Second acquisition unit, for obtaining the essential information of log, the base of the log according to the log index file
This information includes the time section of logger computer system operation situation information or application processing data information.
Second indexing units index log corresponding with the essential information of the log for traversing each journal file
File.
Second recording unit, response needed for the corresponding journal file of essential information for recording indexes to the log
Time.
Second judgment unit, for judging the computer system health information or application processing data letter of the record
The time of breath cuts whether be less than the preset time section.
Second migration units, for the computer system health information or application processing data information when the record
Time cut when being less than the preset time and cutting, will according to the store path and title for presetting the distributed file system
The corresponding log index file of the journal file is migrated to the distributed file system.
In embodiments of the present invention, the computer system in the essential information of the log obtained by second acquisition unit is transported
The time of row condition information or application processing data information cuts, so that it is determined that the size that the time cuts, by journal file corresponding day
Will index file is migrated to the distributed file system.Local space can be reduced in time, to promote local spatial benefit
With rate.
All the embodiments in this specification are described in a progressive manner, same and similar portion between each embodiment
Dividing may refer to each other, and each embodiment focuses on the differences from other embodiments.Especially for system or
For system embodiment, since it is substantially similar to the method embodiment, so describing fairly simple, related place is referring to method
The part of embodiment illustrates.System and system embodiment described above is only schematical, wherein the conduct
The unit of separate part description may or may not be physically separated, component shown as a unit can be or
Person may not be physical unit, it can and it is in one place, or may be distributed over multiple network units.It can root
According to actual need that some or all of the modules therein is selected to achieve the purpose of the solution of this embodiment.Ordinary skill
Personnel can understand and implement without creative efforts.
Professional further appreciates that, unit described in conjunction with the examples disclosed in the embodiments of the present disclosure
And algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware and
The interchangeability of software generally describes each exemplary composition and step according to function in the above description.These
Function is implemented in hardware or software actually, the specific application and design constraint depending on technical solution.Profession
Technical staff can use different methods to achieve the described function each specific application, but this realization is not answered
Think beyond the scope of this invention.
The foregoing description of the disclosed embodiments enables those skilled in the art to implement or use the present invention.
Various modifications to these embodiments will be readily apparent to those skilled in the art, as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, of the invention
It is not intended to be limited to the embodiments shown herein, and is to fit to and the principles and novel features disclosed herein phase one
The widest scope of cause.
Claims (10)
1. a kind of log processing method, which is characterized in that the described method includes:
Response time needed for indexing journal file according to log index file, alternatively, according to recording in log index file
The time of occurrence log cuts, and the log index file for being pre-stored within index database is stored to distributed file system；
Relevant processing operation is executed to the journal file that the log index file indexes according to preset rules.
2. the method according to claim 1, wherein described in advance in the mistake of index database storage log index file
Journey, comprising:
Collector journal, and journal file is established according to Log Types；
According to the essential information of log each in journal file, log index file is established, wherein each in the journal file
The essential information of log include logger computer system operation situation information or application processing data information time cut, serial number and
The size of log；
The log index file is stored to index database.
3. the method according to claim 1, wherein needed for the index journal file according to log index file
Response time, alternatively, according to the time of the occurrence log recorded in log index file cut, index database will be pre-stored within
Log index file, which is stored to distributed file system, includes:
According to the log index file, the essential information of log is obtained, the essential information of the log includes logger computer
The time of system operation situation information or application processing data information cuts；
Each journal file is traversed, journal file corresponding with the essential information of the log is indexed；
Response time needed for the corresponding journal file of the essential information of recording indexes to the log；
It is default whether the response time needed for indexing the corresponding journal file of essential information of log described in judgement is more than or equal to
Response time；
The response time needed for the corresponding journal file of the essential information for indexing log is more than or equal to preset response
When the time, according to the store path and title for presetting the distributed file system, by the journal file corresponding day
Will index file is migrated to the distributed file system.
4. the method according to claim 1, wherein needed for the index journal file according to log index file
Response time, alternatively, according to the time of the occurrence log recorded in log index file cut, index database will be pre-stored within
Log index file, which is stored to distributed file system, includes:
According to the log index file, the essential information of log is obtained, the essential information of the log includes logger computer
The time of system operation situation information or application processing data information cuts；
Each journal file is traversed, journal file corresponding with the essential information of the log is indexed；
Response time needed for the corresponding journal file of the essential information of recording indexes to the log；
The time of the computer system health information or application processing data information that judge the record cuts whether be less than in advance
If time cut；
When the time of the computer system health information of the record or application processing data information, which cuts, is less than preset
Between cut when, according to the store path and title for presetting the distributed file system, by the journal file corresponding day
Will index file is migrated to the distributed file system.
5. the method according to claim 1, wherein it is described according to preset rules to the log index file rope
The journal file guided to executes relevant processing operation, comprising:
External interface is called, data parameters are generated；
Match operating parameter corresponding with the data parameters in preset database；
Filing is executed according to the journal file that the operating parameter indexes the log index file, removes or save behaviour
Make.
6. according to the method described in claim 5, it is characterized in that, the method also includes:
When executing filing to the journal file that the log index file indexes according to the operating parameter or saving operation,
The journal file is filed or is saved to the storage system of predesignated storage location.
7. the method according to claim 1, wherein the method also includes:
According to preset Hyper text transfer security protocol, the journal file and the log index file are shown.
8. a kind of log processing device, which is characterized in that described device includes: log management unit and log indexing units；
The log indexing units, for the response time needed for indexing journal file according to log index file, alternatively, according to
The time of the occurrence log recorded in log index file cut, by the log index file for being pre-stored within index database store to point
Cloth file system；
The log management unit, for executing phase to the journal file that the log index file indexes according to preset rules
The processing operation of pass.
9. the apparatus according to claim 1, which is characterized in that the log management unit is provided with external interface, described
Log management unit includes:
Generation unit generates data parameters for calling external interface；
Matching unit, for matching operating parameter corresponding with the data parameters in preset database；
Execution unit, the journal file for being indexed according to the operating parameter to the log index file execute filing,
Remove or save operation.
10. the apparatus according to claim 1, which is characterized in that described device further include: blog search boundary element；
The blog search boundary element, for according to preset Hyper text transfer security protocol, show the journal file and
The log index file.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910704328.4A CN110515898B (en) | 2019-07-31 | 2019-07-31 | Log processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910704328.4A CN110515898B (en) | 2019-07-31 | 2019-07-31 | Log processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110515898A true CN110515898A (en) | 2019-11-29 |
| CN110515898B CN110515898B (en) | 2022-04-22 |
Family
ID=68624132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910704328.4A Active CN110515898B (en) | 2019-07-31 | 2019-07-31 | Log processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110515898B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113485962A (en) * | 2021-06-30 | 2021-10-08 | 中国民航信息网络股份有限公司 | Log file storage method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073706A (en) * | 2010-12-30 | 2011-05-25 | 北京锐安科技有限公司 | Combined application method of distributed file storage system and relation database |
| CN102096626A (en) * | 2010-12-27 | 2011-06-15 | 意法·爱立信半导体（北京）有限公司 | Mobile terminal and processing method of test log thereof |
| US20140215257A1 (en) * | 2013-01-30 | 2014-07-31 | Ebay Inc | Daisy chain distribution in data centers |
| CN106649461A (en) * | 2016-09-26 | 2017-05-10 | 浪潮电子信息产业股份有限公司 | Method for automatically cleaning and maintaining ElasticSearch log index file |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
-
2019
- 2019-07-31 CN CN201910704328.4A patent/CN110515898B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102096626A (en) * | 2010-12-27 | 2011-06-15 | 意法·爱立信半导体（北京）有限公司 | Mobile terminal and processing method of test log thereof |
| CN102073706A (en) * | 2010-12-30 | 2011-05-25 | 北京锐安科技有限公司 | Combined application method of distributed file storage system and relation database |
| US20140215257A1 (en) * | 2013-01-30 | 2014-07-31 | Ebay Inc | Daisy chain distribution in data centers |
| CN106649461A (en) * | 2016-09-26 | 2017-05-10 | 浪潮电子信息产业股份有限公司 | Method for automatically cleaning and maintaining ElasticSearch log index file |
| CN109933505A (en) * | 2019-03-14 | 2019-06-25 | 深圳市珍爱捷云信息技术有限公司 | Log processing method, device, computer equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113485962A (en) * | 2021-06-30 | 2021-10-08 | 中国民航信息网络股份有限公司 | Log file storage method, device, equipment and storage medium |
| CN113485962B (en) * | 2021-06-30 | 2023-08-01 | 中国民航信息网络股份有限公司 | Log file storage method, device, equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110515898B (en) | 2022-04-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9922114B2 (en) | Systems and methods for distributing indexer configurations | |
| US10282113B2 (en) | Systems and methods for providing a unified view of primary and secondary storage resources | |
| US11301136B2 (en) | Capacity forecasting based on capacity policies and transactions | |
| Klemettinen et al. | Rule discovery in telecommunication alarm data | |
| EP0443974A2 (en) | Method for maintaining a selective document history log in a data processing system | |
| CN107832196A (en) | A kind of monitoring device and monitoring method for real-time logs anomalous content | |
| US9514183B1 (en) | Indexing and searching of large amounts of machine generated data collected from disparate sources | |
| EP2214109A1 (en) | Network storage device collector | |
| CN111106965B (en) | Intelligent log analysis method, tool, equipment and medium for complex system | |
| WO2015148328A1 (en) | System and method for accelerating problem diagnosis in software/hardware deployments | |
| CN107608860A (en) | A kind of method, apparatus, the equipment of error log classification storage | |
| JP2006350464A (en) | Data collection system, data extraction server, data collection method and data collection program | |
| CN110515898A (en) | A kind of log processing method and device | |
| US9916068B1 (en) | Graphical user interface for displaying alarm security level of groups of elements | |
| CN104462967B (en) | The method, apparatus and system of file access pattern | |
| CN108173711A (en) | Enterprises system data exchange monitoring method | |
| CN109886318A (en) | A kind of information processing method, device and computer readable storage medium | |
| JP4754787B2 (en) | In-house information system management device | |
| JP6841228B2 (en) | File information collection system, method and program | |
| JP5603447B2 (en) | Log management system, log management apparatus, and log management method | |
| CN116185785A (en) | Early warning method and device for file abnormal change | |
| US10713226B1 (en) | Managing data using archiving | |
| CN109977693A (en) | A kind of generation method and device of forced symmetric centralization rule | |
| JP2008299496A (en) | Operation simulator and program | |
| US11436122B1 (en) | Key performance indicator recommendations based on relevancy ranking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |