CN107832372A - A kind of log analysis method and system - Google Patents
A kind of log analysis method and system Download PDFInfo
- Publication number
- CN107832372A CN107832372A CN201711027648.8A CN201711027648A CN107832372A CN 107832372 A CN107832372 A CN 107832372A CN 201711027648 A CN201711027648 A CN 201711027648A CN 107832372 A CN107832372 A CN 107832372A
- Authority
- CN
- China
- Prior art keywords
- log
- module
- journal file
- submodule
- daily record
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a kind of log analysis method and system.This method includes：Log acquisition module captures journal file in real time；The journal file that is obtained described in log analysis module analysis simultaneously stores analysis result；As a result the analysis result is showed user by display module.Log analysis method proposed by the present invention can be obtained and analyzed to journal file in real time, solve the problems, such as log analysis timeliness hysteresis.
Description
Technical field
The present embodiments relate to a kind of microcomputer data processing, more particularly to a kind of log analysis method and it is
System.
Background technology
Journal file is produced in machine or application running, technical staff is by recording and analyzing journal file, energy
It is enough to obtain the security information applied in running, abnormal information, system mistake etc..According to journal file, technical staff is not only
It can determine the reason for certain is abnormal using generation, moreover it is possible to which the source code to application is improved.However, this log analysis
Under mode, acquisition log information that technical staff lags forever.
The content of the invention
The present invention provides a kind of log analysis method and system, to solve asking for log analysis timeliness hysteresis in the prior art
Topic.
In a first aspect, the embodiments of the invention provide a kind of log analysis method, including：
Log acquisition module captures journal file in real time；
The journal file that is obtained described in log analysis module analysis simultaneously stores analysis result；
As a result the analysis result is showed user by display module.
Further, the log acquisition module captures journal file and also included in real time：
The log acquisition module is converted to the journal file of acquisition the day for unifying form according to certain rule
Will file；
The log acquisition module compresses the journal file of the unified form, forms log file data bag.
Further, the journal file obtained described in the log analysis module analysis includes：
Daily record monitoring submodule in the log analysis module, monitors the log acquisition module, to what is increased newly in real time
The log file data bag is decompressed, and the journal file that decompression is obtained is transferred to daily record analyzing sub-module；
The daily record analyzing sub-module, the journal file for receiving daily record monitoring submodule transmission are simultaneously analyzed, and by day
Will analysis result is transferred to data storage submodule；
The log analysis result of the data storage submodule reception daily record analyzing sub-module transmission, and according to
Storage rule is set by the log analysis result editor and storage.
Further, analysis result is showed user to include by the result display module：
Acquisition of information submodule in the result display module obtains user's request, and is transferred to information inquiry submodule
Block；
Described information inquiry submodule receives the user's request, is inquired about from the data storage submodule needed for user
Data, and it is transferred to information displaying submodule；
Described information displaying submodule receives the user requested data, and the user requested data editor is formed and analyzed
As a result, user is showed.
Further, the log analysis method also includes：
Acquisition of information submodule in the result display module can also receive the journal file of user's upload, and transmit
Analyzed to the log analysis module.
Further, the daily record analyzing sub-module, receive the journal file of daily record monitoring submodule transmission and divide
Analyse, and log analysis result is transferred into data storage submodule also to include：
The daily record analyzing sub-module uses thread-safe function, while analyzes at least two newly-increased journal files, shape
Into analysis result；
The daily record analyzing sub-module carries out duplicate removal, backup, deletion or the operation of recovery to the journal file.
Further, the data storage submodule receives the log analysis knot of the daily record analyzing sub-module transmission
Fruit, and also include the log analysis result editor and storage according to setting storage rule：
The data storage submodule carries out duplicate removal, backup, deletion or recovery to the log analysis result of reception
Operation.
Further, the analysis result includes following at least one：Security information, abnormal information, warning message.
Second aspect, the embodiments of the invention provide a kind of Log Analysis System, including：
Log acquisition module, for obtaining journal file in real time；
Log analysis module, for analyzing the journal file of the acquisition and storing analysis result；
As a result display module, for analysis result to be showed into user.
The present invention parses by obtaining journal file in real time, and to journal file, can obtain in time in journal file
The machines such as security information, abnormal information or application related information, solves the problems, such as log analysis timeliness hysteresis.
Brief description of the drawings
Fig. 1 is a kind of flow chart of log analysis method in the embodiment of the present invention one.
Fig. 2 is the flow chart of another log analysis method in the embodiment of the present invention two.
Fig. 3 is a kind of structural representation of Log Analysis System in the embodiment of the present invention two.
Fig. 4 is the structural representation of another Log Analysis System in the embodiment of the present invention two.
Embodiment
The present invention is described in further detail with reference to the accompanying drawings and examples.It is understood that this place is retouched
The specific embodiment stated is used only for explaining the present invention, rather than limitation of the invention.It also should be noted that in order to just
Part related to the present invention rather than entire infrastructure are illustrate only in description, accompanying drawing.
Embodiment one
Fig. 1 is a kind of flow chart of log analysis method in the embodiment of the present invention one, and the present embodiment is applicable to analyze
Journal file, specifically comprise the following steps：
Step 110, log acquisition module capture journal file in real time.
Wherein, journal file refers to all machine datas, including directly caused data of machine hardware or on machine
Data caused by the software of operation, for example, technical staff export in a program programming language word, by acting on behalf of taken at regular intervals
Systematic parameter or pass through user behavior data etc. that collector obtains be implanted into webpage.
Wherein, exemplarily, as shown in Fig. 2 capture journal file includes in real time：
The journal file of acquisition is converted to unification by step 111, the log acquisition module according to certain rule
The journal file of form；
Wherein, the acquisition of journal file refer to by network or other approach by journal file as caused by journal file
Position copies local to.
Wherein, because the analysis of information in journal file is that statistics is wherein by the way that journal file and script file are contrasted
What related or similar information obtained, therefore, it is necessary to the journal file tentatively obtained to be converted to the journal file of unified form,
Such as be converted to the journal file of JSON forms.
Step 112, the log acquisition module compress the journal file of the unified form, form log file data
Bag.
Wherein, a variety of condensing routines can be utilized by journal file being compressed, such as compressed package program or press journey soon
Sequence etc., compressing the form of the packet of formation includes .tar forms or tar.gz forms etc..
Wherein, journal file is compressed, the purpose for forming packet is to reduce the space shared by journal file, improves daily record
Transmission rate of the file by log acquisition module transfer to log analysis module.
The journal file that is obtained described in step 120, log analysis module analysis simultaneously stores analysis result.
Exemplarily, as shown in Fig. 2 step 120 specifically includes：
Daily record monitoring submodule in step 121, the log analysis module, monitors the log acquisition module in real time,
The newly-increased log file data bag is decompressed, and the journal file that decompression is obtained is transferred to daily record parsing submodule
Block；
Step 122, the daily record analyzing sub-module, receive the journal file of daily record monitoring submodule transmission and divide
Analysis, and log analysis result is transferred to data storage submodule；
Wherein, journal file is analyzed, including journal file is compared with script file, accounting log file
In the information that includes.
Wherein, log analysis result includes security information, abnormal information or warning message etc..
Step 123, the data storage submodule receive the log analysis knot of the daily record analyzing sub-module transmission
Fruit, and according to default storage rule by the log analysis result editor and storage.
Wherein, by log analysis result according to storage rule editor include to log analysis result carry out with or it is or non-
Deng operation.
Further, the daily record analyzing sub-module uses thread-safe function, while analyzes newly-increased at least two days
Will file, form analysis result；
Further, the daily record analyzing sub-module carries out duplicate removal, backup, deletion or the operation of recovery to journal file.
Wherein, when thread-safe function refers to that daily record analyzing sub-module analyzes at least two journal files at the same time, each day
Will analysis process is not interfere with each other, and ensure that precision of analysis.
Wherein, the backup operation to journal file is in order to save the integrality of log information from damage, when under some storing path
Partial log file for some reason, such as virus is invaded and harassed or when assault etc. is lost, can be in the file of backup
Give for change.
Wherein, the duplicate removal of journal file is included daily record analyzing sub-module detect at least two journal files information whether
It is completely the same, including the ageing information such as time caused by journal file, if so, then deleting the journal file of repetition.
Wherein, the journal file that the recovery to journal file refers to delete by mistake returns to original position.
The analysis result is showed user by step 130, result display module.
Wherein, as a result display module can be web interface or query software interface etc..
Exemplarily, as shown in Fig. 2 step 130 specifically includes：
Acquisition of information submodule in step 131, the result display module obtains user's request, and is transferred to information and looks into
Ask submodule；
Wherein, demand or customer analysis of the user's request including user's query log files analysis result manually import
Demand of journal file etc..
Step 132, described information inquiry submodule receive the user's request, are inquired about from the data storage submodule
User requested data, and it is transferred to information displaying submodule；
Wherein, information inquiry submodule by the comparison of user's demand information and log file analysis result, match, find
Data needed for user, it is transferred to information displaying submodule.
Step 133, described information displaying submodule receive the user requested data, by the user requested data editor
Analysis result is formed, shows user.
Wherein, required data and log file analysis result can by digitlization, tabular or it is patterned in the form of exhibition
Show to user, to user with it is clear, intuitively experience.
The technical scheme of the present embodiment, by monitoring in real time, obtaining journal file, journal file is analyzed and will be divided
Result storage is analysed, so that technical staff or user inquire about so that technical staff or user can be obtained in journal file in time
The machines such as security information, abnormal information or application related information, solves the problems, such as log analysis timeliness hysteresis.
Embodiment two
Fig. 3 is a kind of structural representation for Log Analysis System that the embodiment of the present invention two provides, in above-described embodiment one
On the basis of the log analysis method of offer, the Log Analysis System specifically includes：
Log acquisition module 210, for obtaining journal file in real time；
Log analysis module 220, for for analyzing the journal file of the acquisition and storing analysis result；
As a result display module 230, for analysis result to be showed into user.
Exemplarily, as shown in figure 4, log analysis module 220 specifically includes：
Daily record monitors submodule 221, for monitoring the log acquisition module 210 in real time, to newly-increased daily record text
Part packet is decompressed, and the journal file that decompression is obtained is transferred to daily record analyzing sub-module 222；
Daily record analyzing sub-module 222, for receiving the journal file of the daily record monitoring transmission of submodule 221 and analyzing,
And log analysis result is transferred to data storage submodule 223；
Wherein, journal file is analyzed, including journal file is compared with script file, accounting log file
In the information that includes.
Wherein, log analysis result includes security information, abnormal information or warning message etc..
Data storage submodule 223, the log analysis result that the daily record analyzing sub-module 222 is transmitted is received, and
According to default storage rule by the log analysis result editor and storage.
Wherein, by log analysis result according to storage rule editor include to log analysis result carry out with or it is or non-
Deng operation.
Exemplarily, as shown in figure 4, result display module 230 specifically includes：
Acquisition of information submodule 231, for obtaining user's request, and it is transferred to information inquiry submodule 232；
Wherein, demand or customer analysis of the user's request including user's query log files analysis result manually import
Demand of journal file etc..
Information inquiry submodule 232, the user's request is received, user is inquired about from the data storage submodule 223
Required data, and it is transferred to information displaying submodule 233；
Wherein, information inquiry submodule by the comparison of user's demand information and log file analysis result, match, find
Data needed for user, it is transferred to information displaying submodule.
Information shows submodule 233, receives the user requested data, the user requested data editor is formed
Analysis result, show user.
Wherein, required data and log file analysis result can by digitlization, tabular or it is patterned in the form of exhibition
Show to user, to user with it is clear, intuitively experience.
The technical scheme of the present embodiment, the effect transmitted by each intermodular data, mutually cooperateed with, prison in real time can be achieved
Survey, obtain journal file, journal file is analyzed and stores analysis result, so that technical staff or user inquire about,
Enabling technical staff, either user obtains the machines such as security information in journal file, abnormal information or the related letter of application in time
Breath, solves the problems, such as log analysis timeliness hysteresis.
Pay attention to, above are only presently preferred embodiments of the present invention and institute's application technology principle.It will be appreciated by those skilled in the art that
The invention is not restricted to specific embodiment described here, can carry out for a person skilled in the art various obvious changes,
Readjust and substitute without departing from protection scope of the present invention.Therefore, although being carried out by above example to the present invention
It is described in further detail, but the present invention is not limited only to above example, without departing from the inventive concept, also
Other more equivalent embodiments can be included, and the scope of the present invention is determined by scope of the appended claims.
Claims (9)
- A kind of 1. log analysis method, it is characterised in that including：Log acquisition module captures journal file in real time；The journal file that is obtained described in log analysis module analysis simultaneously stores analysis result；As a result the analysis result is showed user by display module.
- 2. according to the method for claim 1, it is characterised in that the log acquisition module captures journal file and also wrapped in real time Include：For the log acquisition module according to certain rule, the daily record that the journal file of acquisition is converted to unified form is literary Part；The log acquisition module compresses the journal file of the unified form, forms log file data bag.
- 3. according to the method for claim 1, it is characterised in that the daily record text obtained described in the log analysis module analysis Part includes：Daily record monitoring submodule in the log analysis module, monitors the log acquisition module, described in newly-increased in real time Log file data bag is decompressed, and the journal file that decompression is obtained is transferred to daily record analyzing sub-module；The daily record analyzing sub-module, receive the journal file of daily record monitoring submodule transmission and analyze, and daily record is divided Analysis result is transferred to data storage submodule；The data storage submodule receives the log analysis result of the daily record analyzing sub-module transmission, and according to setting Storage rule is by the log analysis result editor and storage.
- 4. the method according to claim 1 or 3, it is characterised in that the result display module shows analysis result User includes：Acquisition of information submodule in the result display module obtains user's request, and is transferred to information inquiry submodule；Described information inquiry submodule receives the user's request, and number needed for user is inquired about from the data storage submodule According to, and it is transferred to information displaying submodule；Described information displaying submodule receives the user requested data, and the user requested data editor is formed into analysis knot Fruit, show user.
- 5. according to the method for claim 4, it is characterised in that also include：Acquisition of information submodule in the result display module can also receive the journal file of user's upload, and be transferred to institute Log analysis module is stated to be analyzed.
- 6. according to the method for claim 3, it is characterised in that the daily record analyzing sub-module, receive daily record monitoring submodule The journal file of block transmission is simultaneously analyzed, and log analysis result is transferred into data storage submodule also included：The daily record analyzing sub-module uses thread-safe function, while analyzes at least two newly-increased journal files, is formed and divided Analyse result；The daily record analyzing sub-module carries out duplicate removal, backup, deletion or the operation of recovery to the journal file.
- 7. according to the method for claim 3, it is characterised in that the data storage submodule receives daily record parsing The log analysis result of module transfer, and also wrap the log analysis result editor and storage according to setting storage rule Include：The data storage submodule carries out duplicate removal, backup, deletion or the operation of recovery to the log analysis result of reception.
- 8. according to the method for claim 4, it is characterised in that the analysis result includes following at least one：Safety letter Breath, abnormal information, warning message.
- A kind of 9. Log Analysis System, it is characterised in that including：Log acquisition module, for obtaining journal file in real time；Log analysis module, for analyzing the journal file of the acquisition and storing analysis result；As a result display module, for the analysis result to be showed into user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711027648.8A CN107832372A (en) | 2017-10-27 | 2017-10-27 | A kind of log analysis method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711027648.8A CN107832372A (en) | 2017-10-27 | 2017-10-27 | A kind of log analysis method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107832372A true CN107832372A (en) | 2018-03-23 |
Family
ID=61649970
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711027648.8A Withdrawn CN107832372A (en) | 2017-10-27 | 2017-10-27 | A kind of log analysis method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107832372A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112380105A (en) * | 2020-11-23 | 2021-02-19 | 华人运通(上海)云计算科技有限公司 | Log collection method, device, system, equipment, storage medium and plug-in |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| US20130073532A1 (en) * | 2011-09-21 | 2013-03-21 | International Business Machines Corporation | Coordination of event logging operations and log management |
| CN103617287A (en) * | 2013-12-12 | 2014-03-05 | 用友软件股份有限公司 | Log management method and device in distributed environment |
| CN104978438A (en) * | 2015-07-23 | 2015-10-14 | 上海斐讯数据通信技术有限公司 | Journal-based real-time analyzing method and system |
| CN106850761A (en) * | 2016-12-30 | 2017-06-13 | 江苏天联信息科技发展有限公司 | Journal file storage method and device |
-
2017
- 2017-10-27 CN CN201711027648.8A patent/CN107832372A/en not_active Withdrawn
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102411533A (en) * | 2011-08-08 | 2012-04-11 | 浪潮电子信息产业股份有限公司 | Log-management optimizing method for clustered storage system |
| US20130073532A1 (en) * | 2011-09-21 | 2013-03-21 | International Business Machines Corporation | Coordination of event logging operations and log management |
| CN103617287A (en) * | 2013-12-12 | 2014-03-05 | 用友软件股份有限公司 | Log management method and device in distributed environment |
| CN104978438A (en) * | 2015-07-23 | 2015-10-14 | 上海斐讯数据通信技术有限公司 | Journal-based real-time analyzing method and system |
| CN106850761A (en) * | 2016-12-30 | 2017-06-13 | 江苏天联信息科技发展有限公司 | Journal file storage method and device |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112380105A (en) * | 2020-11-23 | 2021-02-19 | 华人运通(上海)云计算科技有限公司 | Log collection method, device, system, equipment, storage medium and plug-in |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11238069B2 (en) | Transforming a data stream into structured data | |
| US11615082B1 (en) | Using a data store and message queue to ingest data for a data intake and query system | |
| US11126614B2 (en) | Log query user interface | |
| DE102016119084A1 (en) | Distributed performance monitoring and analysis of industrial plants | |
| US11966797B2 (en) | Indexing data at a data intake and query system based on a node capacity threshold | |
| CN110457190A (en) | A kind of full link monitoring method, apparatus and system based on block chain | |
| CN112559475B (en) | Data real-time capturing and transmitting method and system | |
| US11609913B1 (en) | Reassigning data groups from backup to searching for a processing node | |
| CN106209431A (en) | A kind of Approaches of Alarm Correlation and network management system | |
| CN112181931A (en) | Big data system link tracking method and electronic equipment | |
| CN110932918A (en) | Log data acquisition method and device and storage medium | |
| CN113805777A (en) | Method and system for generating optimal operation path of service system | |
| CN109783330B (en) | Log processing method, log display method, and related device and system | |
| CN111291028A (en) | High-speed industrial field oriented data acquisition system and method | |
| CN106557483B (en) | Data processing method, data query method, data processing equipment and data query equipment | |
| CN104750814B (en) | The automatic storage method of polynary heterogeneous data flow based on multisensor | |
| CN107832372A (en) | A kind of log analysis method and system | |
| US20180295145A1 (en) | Multicomputer Digital Data Processing to Provide Information Security Control | |
| CN116910820A (en) | Data report processing method, device, computer equipment and storage medium | |
| CN115017218B (en) | Processing method and device of distributed call chain, storage medium and electronic equipment | |
| US11734297B1 (en) | Monitoring platform job integration in computer analytics system | |
| CN116506300A (en) | Website traffic data statistics method and system | |
| KR100423149B1 (en) | A system and a method for message statistics | |
| CN106776794B (en) | Mass data processing method and system | |
| KR102433764B1 (en) | System and method for monitoring service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20180323 |