CN110888849B - 一种在线日志解析方法、系统及其电子终端设备 - Google Patents
一种在线日志解析方法、系统及其电子终端设备 Download PDFInfo
- Publication number
- CN110888849B CN110888849B CN201911077285.8A CN201911077285A CN110888849B CN 110888849 B CN110888849 B CN 110888849B CN 201911077285 A CN201911077285 A CN 201911077285A CN 110888849 B CN110888849 B CN 110888849B
- Authority
- CN
- China
- Prior art keywords
- log
- unresolved
- sequence
- template
- group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000004458 analytical method Methods 0.000 title abstract description 31
- 238000000034 method Methods 0.000 claims abstract description 51
- 238000007781 pre-processing Methods 0.000 claims abstract description 14
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000014509 gene expression Effects 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 abstract description 5
- 238000012545 processing Methods 0.000 abstract description 2
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000001514 detection method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004880 explosion Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/1734—Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/31—Indexing; Data structures therefor; Storage structures
- G06F16/316—Indexing structures
- G06F16/322—Trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/34—Browsing; Visualisation therefor
- G06F16/345—Summarisation for human users
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/90335—Query processing
- G06F16/90344—Query processing by using string matching techniques
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Debugging And Monitoring (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Claims (12)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911077285.8A CN110888849B (zh) | 2019-11-06 | 2019-11-06 | 一种在线日志解析方法、系统及其电子终端设备 |
EP20789812.3A EP3846048A1 (en) | 2019-11-06 | 2020-06-29 | Online log analysis method, system, and electronic terminal device thereof |
PCT/CN2020/098701 WO2021088385A1 (zh) | 2019-11-06 | 2020-06-29 | 一种在线日志解析方法、系统及其电子终端设备 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911077285.8A CN110888849B (zh) | 2019-11-06 | 2019-11-06 | 一种在线日志解析方法、系统及其电子终端设备 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN110888849A CN110888849A (zh) | 2020-03-17 |
CN110888849B true CN110888849B (zh) | 2022-07-22 |
Family
ID=69746909
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911077285.8A Active CN110888849B (zh) | 2019-11-06 | 2019-11-06 | 一种在线日志解析方法、系统及其电子终端设备 |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP3846048A1 (zh) |
CN (1) | CN110888849B (zh) |
WO (1) | WO2021088385A1 (zh) |
Families Citing this family (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110888849B (zh) * | 2019-11-06 | 2022-07-22 | 国网上海市电力公司 | 一种在线日志解析方法、系统及其电子终端设备 |
CN111581220A (zh) * | 2020-05-28 | 2020-08-25 | 泰康保险集团股份有限公司 | 用于时间序列数据的存储及检索方法、装置、设备及存储介质 |
CN111832280B (zh) * | 2020-07-09 | 2023-06-30 | 北京奇艺世纪科技有限公司 | 剧本信息处理方法、装置、电子设备及存储介质 |
CN112000806B (zh) * | 2020-08-25 | 2023-06-16 | 携程旅游信息技术(上海)有限公司 | 异常日志监控分析方法、系统、设备及存储介质 |
CN113254438A (zh) * | 2020-11-20 | 2021-08-13 | 云智慧(北京)科技有限公司 | 一种基于树结构的日志解析方法和系统 |
CN112463957B (zh) * | 2020-12-14 | 2023-06-02 | 清华大学 | 非结构化文本日志流的摘要提取方法和装置 |
CN112732655B (zh) * | 2021-01-13 | 2024-02-06 | 北京六方云信息技术有限公司 | 针对无格式日志的在线解析方法及系统 |
CN112882997B (zh) * | 2021-02-19 | 2022-06-07 | 武汉大学 | 一种基于N-gram与频繁模式挖掘的系统日志解析方法 |
CN112883004B (zh) * | 2021-02-24 | 2023-04-07 | 上海浦东发展银行股份有限公司 | 一种基于日志聚合的日志知识库与健康度获取方法及系统 |
CN113535955B (zh) * | 2021-07-16 | 2022-10-28 | 中国工商银行股份有限公司 | 一种日志快速归类方法及装置 |
CN113595787B (zh) * | 2021-07-27 | 2024-03-29 | 招商银行股份有限公司 | 基于日志模板的实时日志自动告警方法、程序及介质 |
CN113590421B (zh) * | 2021-07-27 | 2024-04-26 | 招商银行股份有限公司 | 日志模板提取方法、程序产品及存储介质 |
CN114598597B (zh) * | 2022-02-24 | 2023-12-01 | 烽台科技(北京)有限公司 | 多源日志解析方法、装置、计算机设备及介质 |
CN115185525B (zh) * | 2022-05-17 | 2023-07-18 | 贝壳找房(北京)科技有限公司 | 数据倾斜代码块定位方法、装置、设备及介质 |
JP7466878B2 (ja) | 2022-06-16 | 2024-04-15 | ソフトバンク株式会社 | 情報処理装置、情報処理方法及びプログラム |
CN115017268B (zh) * | 2022-08-04 | 2022-10-11 | 北京航空航天大学 | 一种基于树结构的启发式日志抽取方法及系统 |
CN115543950B (zh) * | 2022-09-29 | 2023-06-16 | 杭州中电安科现代科技有限公司 | 一种日志范化的数据处理系统 |
CN115860836B (zh) * | 2022-12-07 | 2023-09-26 | 广东南粤分享汇控股有限公司 | 一种基于用户行为大数据分析的电商服务推送方法及系统 |
CN117033464B (zh) * | 2023-08-11 | 2024-04-02 | 上海鼎茂信息技术有限公司 | 一种基于聚类的日志并行解析算法及应用 |
CN117407242B (zh) * | 2023-10-10 | 2024-04-05 | 浙江大学 | 基于大语言模型的低成本、零样本的在线日志解析方法 |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106339293A (zh) * | 2016-08-20 | 2017-01-18 | 南京理工大学 | 一种基于签名的日志事件提取方法 |
CN110175158A (zh) * | 2019-05-23 | 2019-08-27 | 湖南大学 | 一种基于向量化的日志模板提取方法和系统 |
Family Cites Families (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10187495B2 (en) * | 2016-09-23 | 2019-01-22 | Entit Software Llc | Identifying problematic messages |
US11113317B2 (en) * | 2016-09-29 | 2021-09-07 | Micro Focus Llc | Generating parsing rules for log messages |
CN109144964A (zh) * | 2018-08-21 | 2019-01-04 | 杭州安恒信息技术股份有限公司 | 基于机器学习的日志解析方法和装置 |
CN109981625B (zh) * | 2019-03-18 | 2021-08-27 | 中国人民解放军陆军炮兵防空兵学院郑州校区 | 一种基于在线层次聚类的日志模板抽取方法 |
CN110888849B (zh) * | 2019-11-06 | 2022-07-22 | 国网上海市电力公司 | 一种在线日志解析方法、系统及其电子终端设备 |
-
2019
- 2019-11-06 CN CN201911077285.8A patent/CN110888849B/zh active Active
-
2020
- 2020-06-29 EP EP20789812.3A patent/EP3846048A1/en not_active Ceased
- 2020-06-29 WO PCT/CN2020/098701 patent/WO2021088385A1/zh unknown
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106339293A (zh) * | 2016-08-20 | 2017-01-18 | 南京理工大学 | 一种基于签名的日志事件提取方法 |
CN110175158A (zh) * | 2019-05-23 | 2019-08-27 | 湖南大学 | 一种基于向量化的日志模板提取方法和系统 |
Non-Patent Citations (2)
Title |
---|
ELK日志分析系统在河钢承钢的应用;李静;《软件应用》;20170803;62页 * |
Structured and Unstructured Big Data Analytics;Suyash Mishra 等;《 2017 International Conference on Current Trends in Computer, Electrical, Electronics and Communication 》;20180906;741-746页 * |
Also Published As
Publication number | Publication date |
---|---|
EP3846048A4 (en) | 2021-07-07 |
CN110888849A (zh) | 2020-03-17 |
EP3846048A1 (en) | 2021-07-07 |
WO2021088385A1 (zh) | 2021-05-14 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN110888849B (zh) | 一种在线日志解析方法、系统及其电子终端设备 | |
US10019492B2 (en) | Stop word identification method and apparatus | |
CN109408578B (zh) | 一种针对异构环境监测数据融合方法 | |
Kobayashi et al. | Towards an NLP-based log template generation algorithm for system log analysis | |
CN113626400A (zh) | 基于日志树和解析树的日志事件提取方法及系统 | |
CN106649557B (zh) | 一种缺陷报告与邮件列表语义关联挖掘方法 | |
CN110633371A (zh) | 一种日志分类方法及系统 | |
CN110659175A (zh) | 日志的主干提取方法、分类方法、设备及存储介质 | |
CN112115965A (zh) | 一种基于svm的被动操作系统识别方法、存储介质及设备 | |
CN115017268B (zh) | 一种基于树结构的启发式日志抽取方法及系统 | |
CN111190873B (zh) | 一种用于云原生系统日志训练的日志模式提取方法及系统 | |
CN114398891B (zh) | 基于日志关键词生成kpi曲线并标记波段特征的方法 | |
CN116318830A (zh) | 一种基于生成对抗网络的日志入侵检测系统 | |
CN113723555A (zh) | 异常数据的检测方法及装置、存储介质、终端 | |
CN113032371A (zh) | 数据库语法分析方法、装置和计算机设备 | |
CN115221012B (zh) | 一种日志的聚类解析方法、装置及设备 | |
CN116578700A (zh) | 日志分类方法、日志分类装置、设备及介质 | |
Zhu et al. | ML-parser: An efficient and accurate online log parser | |
US11822578B2 (en) | Matching machine generated data entries to pattern clusters | |
CN112925874B (zh) | 基于案例标记的相似代码搜索方法及系统 | |
CN116822491A (zh) | 日志解析方法及装置、设备、存储介质 | |
CN114969761A (zh) | 一种基于lda主题特征的日志异常检测方法 | |
CN115221013B (zh) | 一种日志模式的确定方法、装置及设备 | |
CN117873905B (zh) | 一种代码同源检测的方法、装置、设备及介质 | |
CN116049700B (zh) | 基于多模态的运检班组画像生成方法及装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB03 | Change of inventor or designer information |
Inventor after: Wu Jinlong Inventor after: Li Jing Inventor after: Wang Xiulan Inventor after: He Xudong Inventor after: Zhang Luwei Inventor after: Hu Junyi Inventor after: Chen Xiaolu Inventor after: Xie Liyan Inventor after: Fang Xiaorong Inventor after: Zhu Bei Inventor before: Wu Jinlong Inventor before: Li Jing Inventor before: Wang Xiulan Inventor before: He Xudong Inventor before: Zhang Luwei Inventor before: Hu Junyi Inventor before: Chen Xiaolu Inventor before: Xie Liyan Inventor before: Fang Xiaorong Inventor before: Zhu Bei |
|
CB03 | Change of inventor or designer information | ||
GR01 | Patent grant | ||
GR01 | Patent grant |