CN106341257A - Method and tool for customizing log analysis rules and automatically analyzing logs - Google Patents
Method and tool for customizing log analysis rules and automatically analyzing logs Download PDFInfo
- Publication number
- CN106341257A CN106341257A CN201610700651.0A CN201610700651A CN106341257A CN 106341257 A CN106341257 A CN 106341257A CN 201610700651 A CN201610700651 A CN 201610700651A CN 106341257 A CN106341257 A CN 106341257A
- Authority
- CN
- China
- Prior art keywords
- instrument
- canonical
- magic square
- rule
- automatically
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3476—Data logging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N5/00—Computing arrangements using knowledge-based models
- G06N5/02—Knowledge representation; Symbolic representation
- G06N5/022—Knowledge engineering; Knowledge acquisition
- G06N5/025—Extracting rules from data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0677—Localisation of faults
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/875—Monitoring of systems including the internet
Abstract
The invention provides a method and a tool for customizing log analysis rules and automatically analyzing logs. A simple-operation, visual and guide type operation mode facilitating think tank construction is provided for equipment maintenance personnel and equipment manufacturers, and the method and the tool capable of customizing the system log structure and automatically analyzing the logs are provided so that the equipment maintenance personnel are enabled to more conveniently analyze the equipment logs. According to the tool, multiple access interfaces are provided for internal and external systems so as to act as the component of an expert system developed by the equipment maintenance personnel and also act as the component in the expert system provided by the original equipment manufacturer. The tool is suitable for the real-time and quasi-real-time logs and also suitable for various texts (such as TXT, XLS, XLSX and LOG files and relational and non-relational database storage, etc.) and structured and unstructured data.
Description
First, technical field
The analysis parsing of software Development Automation computer network machine intelligence big data and enforcement engine
2nd, background technology
For the purpose simplifying description, herein will be using some special concepts.When present disclosure is related to related notion, involved
And the implication of concept refer to and is defined as below:
[defining 1] instrument
Instrument mentioned here refers to self-defined daily record resolution rules and automatically parses the Method and kit for of daily record.
[defining 2] etc.
Etc. be illustrate item, content comprise but be not limited to this invention scope.
[defining 3] canonical magic square
Canonical magic square is the general designation of the regular expression label that instrument is capable of automatic identification, and instrument internal portion is just commonly used
Then expression formula tag library, user passes through visualized graph interface can self-defined expansion canonical magic square storehouse.For using user,
User is added dynamically to canonical magic square label to prepare in rule, without implementing of care regular expression label.
[defining 4] wizard-like is prepared
Wizard-like is prepared and is referred under visualized operation, and the preparation page elements of next step are the preparation institutes according to previous step
Determine.
With the fast development of computer network and communication network, the production of all trades and professions and the number of devices of management system
Also increase sharply, increased maintenance cost to the system operation of every profession and trade, when certain device node breaks down or during hidden danger, past
The substantial amounts of time go to check toward attendant to be expended and carry out orientation problem with the specifying information in the alarm and daily record of analysis system.
At present, attendant is after receiving fault warning information, and attendant typically first passes through webmaster and checks that equipment is accused
The alert situation with various performance indications, rule of thumb investigates suspected fault point step by step, also needs to logging device simultaneously and passes through people
Machine instruction interaction obtains just finding problem points after logged result is further analyzed again.Generally, attendant is in solution
The time that certainly often first will expend more than 70% when potential faults goes orientation problem, and time-consuming link mainly has man-machine friendship
Mutually, log analysis, data analysis, logic judgment etc..During positioning problems, the log analysis ability of technical staff and experience
Decisive role is served to the time control of issue handling.
In order to be able to lift system daily record analytic ability, it is typically necessary the function of increasing some expert systems in system,
Allow expert system can carry out the function that daily record parses automatically, but the exploitation of current expert system generally requires system equipment producer and props up
Hold, and the built-in expert system daily record analytical capabilities of system suffer from equipment manufacturer and develop the restriction of working days it is impossible in time
Meet the personalization of plant maintenance personnel and interim demand.
Also the daily record analytical tool that some non-original equipment manufacturers provide, such as " daily record is easy " etc. are occurred in that at this stage, but daily record
Easy daily record analytic method is mainly realized by regular expression.Regular expression is that one kind is retouched using single character string
State, mate a series of rules meeting certain grammer, in a lot of text editors, regular expression be usually used to retrieval,
Replace the text that those meet certain pattern.It is intended to through special training using regular expression and have a set threshold, one
As have Basis of Computer Engineering people learn to get up to be easier, but for Non-computer Majors people use just relatively difficult,
If the plant maintenance personnel wanting all trades and professions will learn regular expression and could parse daily record, thus being used for subsequent operation,
So not only workload is big but also more difficult popularization.
In order to allow plant maintenance personnel more easily analyzing device daily record, and it is applied to expert system (outside system
System) intelligent operation in, it is a kind of simple to operate, visual, sharp that we are that plant maintenance personnel and equipment manufacturer provide
In think tank's construction, wizard-like mode of operation, and being capable of self-defined daily record resolution rules the method that automatically parses daily record
And instrument.The internal external system of this instrument provides multiple access interfaces, can be independently developed specially as plant maintenance personnel
Family system assembly it is also possible to as equipment genuine man provide expert system in assembly.
This instrument be applicable not only in real time with quasi real time daily record, apply also for each class text (for example: txt, xls, xlsx,
Log file, relation and non-relational database storage etc.), structuring and unstructured data.
[content of the invention]
The self-defined daily record resolution rules of energy the Method and kit for of parsing daily record automatically, the purpose of its invention is, can be
User provide a kind of simple to operate, visual, be beneficial to think tank's construction, wizard-like mode of operation, and can make by oneself
Adopted daily record resolution rules the Method and kit for of parsing daily record automatically, thus improve the efficiency of daily record parsing.
The chief component of this instrument has: think tank's management module, rule prepare module, condition custom block, canonical
Magic square module, rule parsing engine, regular enforcement engine etc..
1st, think tank's management module
Based on big data analytical calculation, instrument, according to user's history service condition, is automatically advised use than more frequently
Then masterplate and canonical magic square are marked, and the professional knowledge that carries out of intelligence with the shared of experience and reduces repeated construction.
2nd, rule prepares module
Based on the rule preparation of wizard-like, carry out journal formatting configuration by step guiding user and join with daily record resolution rules
Put.During preparing, user can carry out to sample data parsing preview, reduces error, improve operating efficiency and (facilitate user
In real time preparation rule is verified).
3rd, condition custom block
Instrument automatically according to regular masterplate, dynamic load alternative Rule of judgment list.User can be in alternative conditions list
Condition freely combined judgement, with reach parsing logged result purpose.
4th, canonical magic square module
Canonical tag library in canonical magic square, is to have carried out encapsulation and labeling to conventional regular expression, and realizes
Unified management.Canonical magic square is built-in partly to commonly use canonical label, and supports User Defined canonical label.Canonical magic square can spirit
Work is applied to the scene of any text resolution.
5th, rule parsing engine
Automatically the rule template configuring in the 2nd point of parsing, rule template is converted into the execution code of backstage parsing, and will
The regular enforcement engine that code is transmitted at the 6th point goes to execute.
6th, regular enforcement engine
The daily record parsing execution code being generated based on the 5th step, is carried out daily record parsing, and returns analysis result.
[brief description]
Fig. 1 is rule base preparation process schematic diagram:
Fig. 2 is self-defined log-structured and automatically parses the functional frame composition of logging tools:
[specific embodiment]
Below by example in detail specific embodiment, the most frequently used preparation resolution rules are selected to illustrate.
Involved example is only used for parsing the present invention, is not intended to limit the scope of the present invention.The scope of application of the present invention include but not
It is limited to the scope involved by example.
Instantiation mode is related to the process for preparation of 2 scenes: " judgement of bivariate table result " and " coupling keyword judges ".
[preparation process a] self-defined syslog structure-bivariate table result judges
Step 1:
User's newly-built rule masterplate on instrument, enters resolution rules template configuration guide interface.
Step 2:
Preparing resolution rules interface, (acquiescence is not selected, and does not select representative to be not required to format day to select " judgement of bivariate table result "
Will result.)
Step 3:
Prepare interface formatting bivariate table, prepare the rule formatting bivariate table, prepare which row item specifically includes that from
Start parsing, altogether how many row, Column Cata Format, whether to sort result, whether enable senior preparation, effect preview etc..
Step 4:
In judgment rule selection interface, system, according to the selection result of step 2, automatically selects " judging bivariate table result " and makees
For judgment rule.
Step 5:
Preparing bivariate table result interface, preparing the judgment rule of bivariate table result, main item of preparing is being: variable preparation,
Output preparation, analysis result preview etc. when DP is prepared, condition meets.
Step 6:
Preserve warehouse-in and complete the rule masterplate configuration of bivariate table result.
[preparation process b] self-defined syslog structure-coupling keyword judges
Step 1:
User's newly-built rule masterplate on instrument, enters resolution rules template configuration guide interface.
Step 2:
Preparing resolution rules interface, preparing journal formatting rule, do not choose and (give tacit consent to and do not select, do not select representative to be not required to lattice
Formula logged result.)
Step 3:
In judgment rule selection interface, select " coupling keyword judges ".
Step 4:
Judge interface in coupling keyword, prepare coupling keyword judgment rule, main item of preparing is: keyword, sample
Output preparation etc. when data, matching result, condition meet.
Step 5:
Preservation warehouse-in completes coupling keyword and judges that masterplate configures.
Said process refers to Fig. 1.
Claims (6)
1. a kind of self-defined syslog structure is it is characterised in that visual, simple to operate, the construction of dynamic think tank, to
The rule of conduction is prepared, is automatically parsed and execution instrument.
2. visual, wizard-like mode of operation according to claim 1 is it is characterised in that ask according to user service
Ask, prepare wall scroll or the batch Dynamic expansion of rule base and canonical magic square storehouse support inside and outside system, to reach experience and knowledge
Shared.
3. it is characterised in that being asked according to user service, instrument parses configuration automatically for automatic parsing according to claim 1
Template information, configuration template may be from internal configuration rule storehouse, also may be from the request data of external system, once can solve
Analysis one or a collection of configuration data.
4. it is characterised in that being asked according to user service, instrument executes automatically for automatic execution according to claim 1
The configuration template information of parsing, execution may be from internal request, also may be from the request of external system, once can execute one
Bar or a collection of configuration data.
5. think tank according to claim 1 construction is it is characterised in that instrument is according to user's history data, based on data analysis
(general analysis or big data analysis), is automatically marked some using than more frequently regular masterplate and canonical magic square,
Intelligence carry out the shared of professional knowledge and experience.
6. the above may have been used the special concept that some set for the purpose simplifying description, is defined as follows:
[defining 1] instrument
Instrument mentioned here refers to self-defined syslog structure and automatically parses the Method and kit for of daily record.
[defining 2] etc.
Etc. be illustrate item, but content comprise but be not limited to this invention scope.
[defining 3] canonical magic square
Canonical magic square is the general designation of the regular expression label that instrument is capable of automatic identification, and instrument internal portion commonly uses canonical table
Reach formula tag library, user passes through visualized graph interface can self-defined expansion canonical magic square storehouse.For using user, user
Canonical magic square label is added dynamically to prepare in rule, without implementing of care regular expression label.
[defining 4] wizard-like is prepared
Wizard-like is prepared and is referred under visualized operation, and the preparation page elements of next step are to be determined according to the preparation of previous step
's.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610700651.0A CN106341257B (en) | 2016-08-18 | 2016-08-18 | Device for self-defining log analysis rule and automatically analyzing log |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610700651.0A CN106341257B (en) | 2016-08-18 | 2016-08-18 | Device for self-defining log analysis rule and automatically analyzing log |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106341257A true CN106341257A (en) | 2017-01-18 |
| CN106341257B CN106341257B (en) | 2019-12-10 |
Family
ID=57825330
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610700651.0A Active CN106341257B (en) | 2016-08-18 | 2016-08-18 | Device for self-defining log analysis rule and automatically analyzing log |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106341257B (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107919981A (en) * | 2017-10-31 | 2018-04-17 | 江苏省未来网络创新研究院 | A kind of analysis method of multi-vendor log cache |
| CN109271272A (en) * | 2018-10-15 | 2019-01-25 | 江苏物联网研究发展中心 | Big data component faults based on unstructured log assist repair system |
| CN109783330A (en) * | 2018-12-10 | 2019-05-21 | 北京京东金融科技控股有限公司 | Log processing method, display methods and relevant apparatus, system |
| CN110413578A (en) * | 2019-06-28 | 2019-11-05 | 北京互金新融科技有限公司 | The method and apparatus of data parsing |
| CN111125225A (en) * | 2019-12-24 | 2020-05-08 | 北京数衍科技有限公司 | Bill data analysis method and device and server |
| CN111741029A (en) * | 2020-08-25 | 2020-10-02 | 北京安帝科技有限公司 | Log data processing method, processing device and storage medium |
| CN113377419A (en) * | 2021-05-31 | 2021-09-10 | 同盾科技有限公司 | Business processing method and device, readable storage medium and electronic equipment |
| CN115587158A (en) * | 2022-12-08 | 2023-01-10 | 广东名阳信息科技有限公司 | Log data conversion method and system based on visual configuration |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183625A (en) * | 2015-08-31 | 2015-12-23 | 北京奇虎科技有限公司 | Log data processing method and apparatus |
| CN105550378A (en) * | 2016-02-04 | 2016-05-04 | 贵州电网有限责任公司信息中心 | Extraction and analysis method for heterogeneous security log information under complex network system |
-
2016
- 2016-08-18 CN CN201610700651.0A patent/CN106341257B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105183625A (en) * | 2015-08-31 | 2015-12-23 | 北京奇虎科技有限公司 | Log data processing method and apparatus |
| CN105550378A (en) * | 2016-02-04 | 2016-05-04 | 贵州电网有限责任公司信息中心 | Extraction and analysis method for heterogeneous security log information under complex network system |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107919981A (en) * | 2017-10-31 | 2018-04-17 | 江苏省未来网络创新研究院 | A kind of analysis method of multi-vendor log cache |
| CN109271272A (en) * | 2018-10-15 | 2019-01-25 | 江苏物联网研究发展中心 | Big data component faults based on unstructured log assist repair system |
| CN109271272B (en) * | 2018-10-15 | 2022-05-17 | 江苏物联网研究发展中心 | Big data assembly fault auxiliary repair system based on unstructured log |
| CN109783330A (en) * | 2018-12-10 | 2019-05-21 | 北京京东金融科技控股有限公司 | Log processing method, display methods and relevant apparatus, system |
| CN109783330B (en) * | 2018-12-10 | 2023-04-07 | 京东科技控股股份有限公司 | Log processing method, log display method, and related device and system |
| CN110413578A (en) * | 2019-06-28 | 2019-11-05 | 北京互金新融科技有限公司 | The method and apparatus of data parsing |
| CN111125225A (en) * | 2019-12-24 | 2020-05-08 | 北京数衍科技有限公司 | Bill data analysis method and device and server |
| CN111741029A (en) * | 2020-08-25 | 2020-10-02 | 北京安帝科技有限公司 | Log data processing method, processing device and storage medium |
| CN111741029B (en) * | 2020-08-25 | 2020-12-04 | 北京安帝科技有限公司 | Log data processing method, processing device and storage medium |
| CN113377419A (en) * | 2021-05-31 | 2021-09-10 | 同盾科技有限公司 | Business processing method and device, readable storage medium and electronic equipment |
| CN115587158A (en) * | 2022-12-08 | 2023-01-10 | 广东名阳信息科技有限公司 | Log data conversion method and system based on visual configuration |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106341257B (en) | 2019-12-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106341257A (en) | Method and tool for customizing log analysis rules and automatically analyzing logs | |
| KR102483025B1 (en) | Operational maintenance systems and methods | |
| CN108351636B (en) | Engineering design tool, system and module | |
| CN102135938B (en) | Software product testing method and system | |
| CN1690684B (en) | Methods and systems for monitoring and diagnosing machinery | |
| CN106777101B (en) | Data processing engine | |
| US8731998B2 (en) | Three dimensional visual representation for identifying problems in monitored model oriented business processes | |
| CN109902105A (en) | For the data query system of micro services framework, method, equipment and storage medium | |
| US7926024B2 (en) | Method and apparatus for managing complex processes | |
| WO2001073546A2 (en) | Industrial automation system graphical programming language storage and transmission | |
| CN108255837B (en) | SQL parser and method | |
| Rogers et al. | GALEN ten years on: Tasks and supporting tools | |
| CN110689148A (en) | Metering equipment fault detection method based on AR technology | |
| CN111813661A (en) | Global service data drive automatic test method, device, equipment and medium | |
| CN103870382A (en) | Code risk detection method and device | |
| Smajevic et al. | From conceptual models to knowledge graphs: a generic model transformation platform | |
| US20090063395A1 (en) | Mapping log sets between different log analysis tools in a problem determination environment | |
| Vasyliuk et al. | Construction Features of the Industrial Environment Control System. | |
| CN105632248A (en) | Safety monitoring system and data processing method therefor | |
| CN113962597A (en) | Data analysis method and device, electronic equipment and storage medium | |
| JP2010140407A (en) | Source code inspection device | |
| CN110956030A (en) | Method and system for comparing configuration information of remote machine of transformer substation | |
| Yusuf et al. | An automatic approach to measure and visualize coupling in object-oriented programs | |
| CN111158653B (en) | SQL language-based integrated development and execution system for real-time computing program | |
| CN111414632B (en) | AST self-synthesis-based embedded program data stream security verification method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| CB03 | Change of inventor or designer information |
Inventor after: Xiao Liangqing Inventor before: Chen Chen |
|
| COR | Change of bibliographic data | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20170119 Address after: Yuexiu District West Road, Guangzhou city of Guangdong Province in 510080. 3, 4 Applicant after: GUANGZHOU HENGHAO DATA TECHNOLOGY CO., LTD. Address before: Yuexiu District West Road, Guangzhou city of Guangdong Province in 510080. 3, 4 Applicant before: Chen Chen |
|
| CB02 | Change of applicant information |
Address after: 510000 B03, E-PARK Creative Park, yuzhuzhi Valley, 32, Keng Kong Street, Maogang village, Whampoa, Guangzhou, Guangdong Applicant after: GUANGZHOU HENGHAO DATA TECHNOLOGY CO., LTD. Address before: 510080 4, No. 3, Ho Group West Road, Yuexiu District, Guangzhou, Guangdong. Applicant before: GUANGZHOU HENGHAO DATA TECHNOLOGY CO., LTD. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |