CN111209258A - Tax end system log real-time analysis method, equipment, medium and system - Google Patents
Tax end system log real-time analysis method, equipment, medium and system Download PDFInfo
- Publication number
- CN111209258A CN111209258A CN201911408600.0A CN201911408600A CN111209258A CN 111209258 A CN111209258 A CN 111209258A CN 201911408600 A CN201911408600 A CN 201911408600A CN 111209258 A CN111209258 A CN 111209258A
- Authority
- CN
- China
- Prior art keywords
- log
- log data
- real
- time
- analysis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000010223 real-time analysis Methods 0.000 title claims abstract description 20
- 238000004458 analytical method Methods 0.000 claims abstract description 60
- 238000012545 processing Methods 0.000 claims abstract description 33
- 238000004364 calculation method Methods 0.000 claims description 13
- 238000005192 partition Methods 0.000 claims description 10
- 238000012423 maintenance Methods 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 5
- 238000004140 cleaning Methods 0.000 claims description 4
- 230000002688 persistence Effects 0.000 claims description 3
- 230000002085 persistent effect Effects 0.000 claims description 3
- 238000013480 data collection Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000007480 spreading Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/14—Details of searching files based on file metadata
- G06F16/156—Query results presentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q40/00—Finance; Insurance; Tax strategies; Processing of corporate or income taxes
- G06Q40/10—Tax strategies
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Business, Economics & Management (AREA)
- Accounting & Taxation (AREA)
- Development Economics (AREA)
- Finance (AREA)
- Marketing (AREA)
- Technology Law (AREA)
- Economics (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Library & Information Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method, equipment, a medium and a system for real-time analysis of a tax end system log. The method comprises the following steps: performing multithreading distributed real-time acquisition on log data generated by the business operation of the tax end system; caching the collected log data into a message queue cluster, and simultaneously performing distributed storage on the collected log data as original log data; extracting log data in the message queue cluster, analyzing and processing the log data in real time based on a stream computing tool, and simultaneously storing an analysis result in real time; and providing an operation interface, and performing query, query result display and offline analysis processing on the original log data based on the big data computing engine through the operation interface, or performing real-time query and query result display on the analysis result stored in real time through the operation interface according to the query condition. The system can be rapidly and accurately obtained, and the system can be rapidly positioned.
Description
Technical Field
The invention relates to the field of data processing, in particular to a method, equipment, a medium and a system for analyzing logs of a tax end system in real time.
Background
The tax office end of the anti-counterfeiting tax control system can generate a large amount of logs in the operation process, log data of each operation and the like of each taxpayer can be recorded, and the logs are stored locally in a file form. After a problem is found, when a developer or an operation and maintenance person checks the problem, each file needs to be checked one by one to search log data related to the problem, and sometimes it needs to count how many operations are performed by a taxpayer identification number in a certain time period, or how many times a certain service, such as a host machine fare collection service, is executed in a certain time period. Sometimes, the log data are disordered and may not be in one file, so that a worker needs to check the log files one by one to analyze and count, a large amount of time is spent, and the working efficiency is seriously influenced.
Therefore, a new tax end log analysis method needs to be provided, which can quickly and accurately acquire relevant log data information, so as to quickly locate system problems and improve working efficiency.
Disclosure of Invention
The invention aims to provide a method, equipment, a medium and a system for analyzing logs of a tax end system in real time, which can quickly and accurately acquire related log data information, so that the system problem can be quickly positioned and the working efficiency can be improved.
In order to achieve the purpose, the invention provides a real-time analysis method for a tax end system log, which comprises the following steps:
step 1: performing multithreading distributed real-time acquisition on log data generated by the business operation of the tax end system;
step 2: caching the collected log data into a message queue cluster, and simultaneously performing distributed storage on the collected log data as original log data;
and step 3: extracting the log data in the message queue cluster, analyzing and processing the log data in real time based on a stream computing tool, and simultaneously storing an analysis result in real time;
and 4, step 4: and providing an operation interface, and performing query, query result display and offline analysis processing on the original log data based on a big data computing engine through the operation interface, or performing real-time query and query result display on the analysis result stored in real time through the operation interface according to query conditions.
Optionally, the step 1 includes: at least one flash log collection client is deployed on the tax end server, and the flash log collection client collects log data of multiple threads of the tax end system based on the configuration file.
Optionally, the step 2 includes: caching the log data acquired in real time through a Kafka message queue cluster, and meanwhile, storing the acquired log data into a Hadoop distributed file system to be used as original log data for persistence and maintenance;
and setting an automatic cleaning mechanism for the Kafka message queue cluster to clean out expired log data.
Optionally, the caching the collected log data by the Kafka message queue cluster includes: and caching the log data through the partition created by the Kafka message queue cluster and the message category associated with the log data.
Optionally, the step 3 includes: and pulling the log data cached in the Kafka message queue cluster through a Spark stream real-time calculation program to analyze so as to obtain complete information of each business operation, outputting an analysis result in real time, and storing the analysis result.
Optionally, in the step 4, the performing, by the big-data-based computing engine, offline analysis processing on the raw log data includes: and performing offline analysis processing on the original log data by using a Spark calculation engine through the operation interface.
The present invention also proposes an electronic device, comprising:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described tax end system log real-time analysis method.
The present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the above tax end system log real-time analysis method.
The invention also provides a real-time analysis system for the tax end system logs, which comprises the following steps:
the log acquisition module is used for acquiring log data generated by each business operation of the tax end system in real time in a multithreading manner;
the log caching module is used for caching the collected log data into a message queue cluster;
the log storage module is used for performing distributed storage on the acquired log data as original log data;
the log analysis module is used for extracting the log data in the message queue cluster, analyzing and processing the log data in real time based on a stream calculation tool and storing an analysis result in real time;
and the log display module is used for providing an operation interface, and performing query, query result display and offline analysis processing on the original log data through the operation interface based on a big data computing engine, or performing real-time query and query result display on the analysis result stored in real time through the operation interface according to query conditions.
Optionally, the log collection module includes at least one Flume log collection client, and the Flume log collection client performs multi-threaded log data collection on the tax end system based on the configuration file;
the log caching module comprises a Kafka message queue cluster, and the Kafka message queue cluster is used for caching the log data acquired in real time;
the log storage module comprises a Hadoop distributed file system, and the Hadoop distributed file system is used for carrying out persistent distributed storage on the acquired log data as original log data;
the log analysis module comprises a Spark stream real-time calculation program and a Spark calculation engine, wherein the Spark stream real-time calculation program is used for pulling the log data cached in the Kafka message queue cluster for analysis so as to obtain the complete information of each business operation, and outputting the analysis result in real time; the Spark calculation engine is used for performing offline analysis processing on the original log data.
The invention has the beneficial effects that:
reading a target log through multiple threads, transmitting the read log data to a message queue cluster for caching the log data, extracting and analyzing the cached log data based on a stream computing tool, outputting a log analysis result in real time, storing the analysis result, and acquiring and storing complete information of each service; the big data computing engine is used for analyzing the original log data and inquiring log analysis results in real time according to different inquiry conditions, so that relevant information can be accurately and quickly obtained, and system problems can be quickly positioned.
The apparatus of the present invention has other features and advantages which will be apparent from or are set forth in detail in the accompanying drawings and the following detailed description, which are incorporated herein, and which together serve to explain certain principles of the invention.
Drawings
The above and other objects, features and advantages of the present invention will become more apparent by describing in more detail exemplary embodiments thereof with reference to the attached drawings, in which like reference numerals generally represent like parts.
Fig. 1 is a flowchart illustrating a method for real-time analyzing logs of a tax end system according to the present invention.
FIG. 2 shows a tax end system log real-time analysis system architecture diagram according to one embodiment of the invention.
Detailed Description
The invention will be described in more detail below with reference to the accompanying drawings. While the preferred embodiments of the present invention are shown in the drawings, it should be understood that the present invention may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.
Fig. 1 is a flowchart illustrating a method for real-time analyzing logs of a tax end system according to the present invention.
As shown in fig. 1, a method for real-time analyzing logs of a tax end system according to the present invention includes:
step 1: performing multithreading distributed real-time acquisition on log data generated by the business operation of the tax end system;
step 2: caching the collected log data into a message queue cluster, and simultaneously performing distributed storage on the collected log data as original log data;
and step 3: extracting log data in the message queue cluster, analyzing and processing the log data in real time based on a stream computing tool, and simultaneously storing an analysis result in real time;
and 4, step 4: and providing an operation interface, and performing query, query result display and offline analysis processing on the original log data based on the big data computing engine through the operation interface, or performing real-time query and query result display on the analysis result stored in real time through the operation interface according to the query condition.
Specifically, the method comprises the steps of reading a multithread target log, transmitting the read log data to a message queue cluster for caching the log data, extracting and analyzing the cached log data based on a flow calculation tool, outputting a log analysis result in real time, storing the analysis result, and acquiring and storing complete information of each service; the big data computing engine is used for analyzing the original log data and inquiring log analysis results in real time according to different inquiry conditions, so that relevant information can be accurately and quickly obtained, and system problems can be quickly positioned.
In this embodiment, at least one Flume log collection client is deployed on the tax end server, and the Flume log collection client performs multithreading log data collection on the tax end system based on the configuration file. And caching the log data acquired in real time through the Kafka message queue cluster, and storing the acquired log data into a Hadoop distributed file system to be used as original log data for persistence. Wherein the log data is cached by the partition created by the Kafka message queue cluster and the message class associated with the log data. And an automatic cleaning mechanism is arranged on the Kafka message queue cluster so as to clean out expired log data.
Specifically, a plurality of flash clients, Kafka clusters and the like are deployed on the tax side server. Reading a multithread target log by configuring a flash configuration file, and transmitting the read log data to a Kafka cluster, wherein the Kafka cluster caches the log data through a created partition and a message type (Topic) associated with the log data; and a Kafka automatic cleaning mechanism is configured to clean out expired log data. In the log storage module, log data is durably stored in a Hadoop Distributed File System (HDFS), and a log stored by the HDFS is set to be original log information.
In this embodiment, step 3 includes: and pulling the log data cached in the Kafka message queue cluster for analysis through a Spark stream real-time calculation program to obtain complete information of each business operation, outputting an analysis result in real time, and storing the analysis result.
Specifically, the log data is transmitted to a Kafka message queue, the log information in Kafka is pulled by a Spark stream (Spark Streaming) real-time calculation program for analysis processing, the log processing information is output in real time, and the processing result is stored.
In this embodiment, in step 4, performing offline analysis processing on the raw log data based on the big data computing engine includes: and performing offline analysis processing on the original log data by using a Spark calculation engine through an operation interface.
Specifically, in the log display module, Spark can be used to perform offline analysis processing on the raw data to obtain a relevant result that the worker wants to obtain, and meanwhile, the worker is supported to query data in real time according to different conditions, and the result is displayed to the worker through different display methods.
An embodiment of the present invention further provides an electronic device, where the electronic device includes:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method for real-time analysis of tax end system logs described above.
The embodiment of the present invention further provides a non-transitory computer-readable storage medium, which stores computer instructions for causing a computer to execute the above real-time analysis method for tax end system logs.
FIG. 2 shows a tax end system log real-time analysis system architecture diagram according to one embodiment of the invention.
As shown in fig. 2, an embodiment of the present invention further provides a real-time analysis system for logs of a tax end system, including:
the log acquisition module is used for acquiring log data generated by each business operation of the tax end system in real time in a multithreading manner;
the log caching module is used for caching the collected log data into the message queue cluster;
the log storage module is used for performing distributed storage on the acquired log data as original log data;
the log analysis module is used for extracting log data in the message queue cluster, analyzing and processing the log data in real time based on a stream calculation tool and storing an analysis result in real time;
and the log display module is used for providing an operation interface, and inquiring, inquiring result displaying and off-line analysis processing are carried out on the original log data through the operation interface based on the big data computing engine, or real-time inquiring and inquiring result displaying are carried out on the analysis result stored in real time through the operation interface according to the inquiring condition.
In this embodiment, the log collection module includes at least one Flume log collection client, and the Flume log collection client performs multithreading log data collection on the tax end system based on the configuration file;
the log caching module comprises a Kafka message queue cluster, and the Kafka message queue cluster is used for caching the log data acquired in real time;
the log storage module comprises a Hadoop distributed file system, and the Hadoop distributed file system is used for carrying out persistent distributed storage on the collected log data as original log data;
the log analysis module comprises a Spark stream real-time calculation program and a Spark calculation engine, wherein the Spark stream real-time calculation program is used for pulling log data cached in the Kafka message queue cluster for analysis so as to obtain complete information of each service operation, and outputting an analysis result in real time; the Spark calculation engine is used for performing offline analysis processing on the raw log data.
Particularly, the flash is a high-availability, high-reliability and distributed system for acquiring, aggregating and transmitting mass logs, and supports various data senders customized in the log system for collecting data; at the same time, flash provides the ability to simply process data and write to various data recipients (customizable).
Kafka is a high throughput distributed publish-subscribe messaging system that can handle all action flow data in a consumer-scale website. The purpose of Kafka is to unify online and offline message processing by a Hadoop parallel load mechanism and provide real-time messages through clustering. The Kafka architecture includes: category (Topic), Partition (Partition), message Producer (Producer), message Consumer (Consumer); in general, one common workflow is Kafka's producer writing a message to topic and consumer reading the message from topic. topic is associated with a log, which is a data structure stored on system disk, and Kafka appends a producer's record to the end of the topic log. the topic log consists of many partitions distributed over multiple files, which may be distributed over multiple Kafka cluster nodes. Kafka distributes topic log partitions across different nodes of a cluster to achieve high performance with horizontal scalability. Spreading partitions facilitate fast writing of data, and Kafka copies partitions to many nodes to provide failover.
Spark is a fast and general computing engine designed specially for large-scale data processing, Spark streaming is a framework constructed on Spark for processing streaming data, the basic principle is to divide the streaming data into small time slices (several seconds), and process the small data in a manner similar to batch processing, and the method is an extension to Spark core API, so as to support expandable, high-throughput and fault-tolerant streaming processing of real-time data streams.
Meanwhile, the system also needs to be matched with a distributed application program coordination service component (Zookeeper), the Zookeeper is software for providing a consistent service for the distributed application, and the provided functions comprise: configuration maintenance, domain name service, distributed synchronization, group service, etc.
Flume, Kafka, Spark and Zookeeper are prior art and will not be described herein.
In summary, the present invention utilizes big data technology, obtains all log information related to each service by analyzing logs in real time, and can filter and store according to different conditions; analyzing historical data by utilizing Spark, and inquiring according to different inquiry conditions, such as the total times of executing services such as paper invoice application and the like by a taxpayer in a time range; and the information can be displayed to a developer or an operation and maintenance person through a page. Not only can the operation and maintenance personnel locate the system problem in time, but also can make relevant measures in time when the abnormity occurs. The time and energy of workers are saved to a great extent, and the working efficiency of enterprises is improved.
Having described embodiments of the present invention, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.
Claims (10)
1. A real-time analysis method for tax end system logs is characterized by comprising the following steps:
step 1: performing multithreading distributed real-time acquisition on log data generated by the business operation of the tax end system;
step 2: caching the collected log data into a message queue cluster, and simultaneously performing distributed storage on the collected log data as original log data;
and step 3: extracting the log data in the message queue cluster, analyzing and processing the log data in real time based on a stream computing tool, and simultaneously storing an analysis result in real time;
and 4, step 4: and providing an operation interface, and performing query, query result display and offline analysis processing on the original log data based on a big data computing engine through the operation interface, or performing real-time query and query result display on the analysis result stored in real time through the operation interface according to query conditions.
2. The tax end system log real-time analysis method according to claim 1, wherein the step 1 comprises:
at least one flash log collection client is deployed on the tax end server, and the flash log collection client collects log data of multiple threads of the tax end system based on the configuration file.
3. The tax end system log real-time analysis method according to claim 1, wherein the step 2 comprises:
caching the log data acquired in real time through a Kafka message queue cluster, and meanwhile, storing the acquired log data into a Hadoop distributed file system to be used as original log data for persistence and maintenance;
and setting an automatic cleaning mechanism for the Kafka message queue cluster to clean out expired log data.
4. The tax end system log real-time analysis method according to claim 3, wherein the caching the collected log data by Kafka message queue clustering comprises:
and caching the log data through the partition created by the Kafka message queue cluster and the message category associated with the log data.
5. The tax end system log real-time analysis method according to claim 1, wherein the step 3 comprises:
and pulling the log data cached in the Kafka message queue cluster through a Spark stream real-time calculation program to analyze so as to obtain complete information of each business operation, outputting an analysis result in real time, and storing the analysis result.
6. The tax end system log real-time analysis method according to claim 1, wherein in the step 4, the performing of the offline analysis processing on the raw log data by the big data computing engine comprises:
and performing offline analysis processing on the original log data by using a Spark calculation engine through the operation interface.
7. An electronic device, characterized in that the electronic device comprises:
at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of real-time tax end system log analysis of any of claims 1-6.
8. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of real-time tax end system log analysis according to any of claims 1-6.
9. A tax end system log real-time analysis system, comprising:
the log acquisition module is used for acquiring log data generated by each business operation of the tax end system in real time in a multithreading manner;
the log caching module is used for caching the collected log data into a message queue cluster;
the log storage module is used for performing distributed storage on the acquired log data as original log data;
the log analysis module is used for extracting the log data in the message queue cluster, analyzing and processing the log data in real time based on a stream calculation tool and storing an analysis result in real time;
and the log display module is used for providing an operation interface, and performing query, query result display and offline analysis processing on the original log data through the operation interface based on a big data computing engine, or performing real-time query and query result display on the analysis result stored in real time through the operation interface according to query conditions.
10. The tax end system log real-time analysis system according to claim 9,
the log acquisition module comprises at least one flash log acquisition client, and the flash log acquisition client acquires log data of multiple threads of the tax end system based on the configuration file;
the log caching module comprises a Kafka message queue cluster, and the Kafka message queue cluster is used for caching the log data acquired in real time;
the log storage module comprises a Hadoop distributed file system, and the Hadoop distributed file system is used for carrying out persistent distributed storage on the acquired log data as original log data;
the log analysis module comprises a Spark stream real-time calculation program and a Spark calculation engine, wherein the Spark stream real-time calculation program is used for pulling the log data cached in the Kafka message queue cluster for analysis so as to obtain the complete information of each business operation, and outputting the analysis result in real time; the Spark calculation engine is used for performing offline analysis processing on the original log data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911408600.0A CN111209258A (en) | 2019-12-31 | 2019-12-31 | Tax end system log real-time analysis method, equipment, medium and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911408600.0A CN111209258A (en) | 2019-12-31 | 2019-12-31 | Tax end system log real-time analysis method, equipment, medium and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111209258A true CN111209258A (en) | 2020-05-29 |
Family
ID=70784161
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911408600.0A Pending CN111209258A (en) | 2019-12-31 | 2019-12-31 | Tax end system log real-time analysis method, equipment, medium and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111209258A (en) |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111897704A (en) * | 2020-06-28 | 2020-11-06 | 杭州涂鸦信息技术有限公司 | Session log analysis method, electronic device and storage medium |
| CN111949633A (en) * | 2020-08-03 | 2020-11-17 | 杭州电子科技大学 | ICT system operation log analysis method based on parallel stream processing |
| CN111984433A (en) * | 2020-07-31 | 2020-11-24 | 五八有限公司 | Business data processing method, display method, device, electronic equipment and medium |
| CN112115112A (en) * | 2020-08-10 | 2020-12-22 | 上海金仕达软件科技有限公司 | Log information processing method and device and electronic equipment |
| CN112202607A (en) * | 2020-09-28 | 2021-01-08 | 中移（杭州）信息技术有限公司 | Statistical calculation method of log message, server and storage medium |
| CN112269808A (en) * | 2020-11-17 | 2021-01-26 | 携程旅游网络技术（上海）有限公司 | Engine query control method, system, device and storage medium |
| CN112860195A (en) * | 2021-03-26 | 2021-05-28 | 中国工商银行股份有限公司 | Log storage method and device |
| CN112860651A (en) * | 2021-02-23 | 2021-05-28 | 北京明略昭辉科技有限公司 | Task log analysis method, system, computer device and storage medium |
| CN113190524A (en) * | 2021-04-21 | 2021-07-30 | 武汉卓尔信息科技有限公司 | Industrial big data acquisition method and system |
| WO2021151315A1 (en) * | 2020-06-29 | 2021-08-05 | 平安科技（深圳）有限公司 | Application system log data processing method, application system, device, and medium |
| CN113282611A (en) * | 2021-06-29 | 2021-08-20 | 深圳平安智汇企业信息管理有限公司 | Method and device for synchronizing stream data, computer equipment and storage medium |
| CN113852664A (en) * | 2021-08-19 | 2021-12-28 | 天津市普迅电力信息技术有限公司 | Energy commodity and energy demand accurate pushing method based on distributed real-time calculation |
| CN114168624A (en) * | 2021-12-08 | 2022-03-11 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
| CN114490744A (en) * | 2021-12-13 | 2022-05-13 | 中企云链（北京）金融信息服务有限公司 | Data caching method, storage medium and electronic device |
| CN114969083A (en) * | 2022-06-24 | 2022-08-30 | 在线途游（北京）科技有限公司 | Real-time data analysis method and system |
| CN115391429A (en) * | 2022-09-14 | 2022-11-25 | 武汉奋进智能机器有限公司 | Time sequence data processing method and device based on big data cloud computing |
| CN115714718A (en) * | 2022-09-23 | 2023-02-24 | 上海芯赛云计算科技有限公司 | Log early warning method and system based on memory, computer equipment and storage medium |
| CN115733718A (en) * | 2021-08-26 | 2023-03-03 | 北京神州泰岳软件股份有限公司 | Weblog analysis method and device, electronic equipment and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105207826A (en) * | 2015-10-26 | 2015-12-30 | 南京联成科技发展有限公司 | Security attack alarm positioning system based on Spark big data platform of Tachyou |
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| US20180129579A1 (en) * | 2016-11-10 | 2018-05-10 | Nec Laboratories America, Inc. | Systems and Methods with a Realtime Log Analysis Framework |
| US20180341956A1 (en) * | 2017-05-26 | 2018-11-29 | Digital River, Inc. | Real-Time Web Analytics System and Method |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
-
2019
- 2019-12-31 CN CN201911408600.0A patent/CN111209258A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105207826A (en) * | 2015-10-26 | 2015-12-30 | 南京联成科技发展有限公司 | Security attack alarm positioning system based on Spark big data platform of Tachyou |
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
| US20180129579A1 (en) * | 2016-11-10 | 2018-05-10 | Nec Laboratories America, Inc. | Systems and Methods with a Realtime Log Analysis Framework |
| CN106709003A (en) * | 2016-12-23 | 2017-05-24 | 长沙理工大学 | Hadoop-based mass log data processing method |
| US20180341956A1 (en) * | 2017-05-26 | 2018-11-29 | Digital River, Inc. | Real-Time Web Analytics System and Method |
| CN109542733A (en) * | 2018-12-05 | 2019-03-29 | 焦点科技股份有限公司 | A kind of highly reliable real-time logs collection and visual m odeling technique method |
Non-Patent Citations (2)
| Title |
|---|
| 杨立鹏等: "基于Storm实时流式计算框架的网络日志分析方法" * |
| 陈斌: "大数据支持下的网络日志分析技术研究" * |
Cited By (23)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111897704A (en) * | 2020-06-28 | 2020-11-06 | 杭州涂鸦信息技术有限公司 | Session log analysis method, electronic device and storage medium |
| WO2021151315A1 (en) * | 2020-06-29 | 2021-08-05 | 平安科技（深圳）有限公司 | Application system log data processing method, application system, device, and medium |
| CN111984433A (en) * | 2020-07-31 | 2020-11-24 | 五八有限公司 | Business data processing method, display method, device, electronic equipment and medium |
| CN111949633A (en) * | 2020-08-03 | 2020-11-17 | 杭州电子科技大学 | ICT system operation log analysis method based on parallel stream processing |
| CN112115112A (en) * | 2020-08-10 | 2020-12-22 | 上海金仕达软件科技有限公司 | Log information processing method and device and electronic equipment |
| CN112202607B (en) * | 2020-09-28 | 2022-06-14 | 中移（杭州）信息技术有限公司 | Statistical calculation method of log message, server and storage medium |
| CN112202607A (en) * | 2020-09-28 | 2021-01-08 | 中移（杭州）信息技术有限公司 | Statistical calculation method of log message, server and storage medium |
| CN112269808A (en) * | 2020-11-17 | 2021-01-26 | 携程旅游网络技术（上海）有限公司 | Engine query control method, system, device and storage medium |
| CN112269808B (en) * | 2020-11-17 | 2024-03-19 | 携程旅游网络技术（上海）有限公司 | Engine query control method, system, equipment and storage medium |
| CN112860651A (en) * | 2021-02-23 | 2021-05-28 | 北京明略昭辉科技有限公司 | Task log analysis method, system, computer device and storage medium |
| CN112860195A (en) * | 2021-03-26 | 2021-05-28 | 中国工商银行股份有限公司 | Log storage method and device |
| CN113190524A (en) * | 2021-04-21 | 2021-07-30 | 武汉卓尔信息科技有限公司 | Industrial big data acquisition method and system |
| CN113282611A (en) * | 2021-06-29 | 2021-08-20 | 深圳平安智汇企业信息管理有限公司 | Method and device for synchronizing stream data, computer equipment and storage medium |
| CN113282611B (en) * | 2021-06-29 | 2024-04-23 | 深圳平安智汇企业信息管理有限公司 | Method, device, computer equipment and storage medium for synchronizing stream data |
| CN113852664A (en) * | 2021-08-19 | 2021-12-28 | 天津市普迅电力信息技术有限公司 | Energy commodity and energy demand accurate pushing method based on distributed real-time calculation |
| CN115733718A (en) * | 2021-08-26 | 2023-03-03 | 北京神州泰岳软件股份有限公司 | Weblog analysis method and device, electronic equipment and storage medium |
| CN114168624A (en) * | 2021-12-08 | 2022-03-11 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
| CN114168624B (en) * | 2021-12-08 | 2022-09-20 | 掌阅科技股份有限公司 | Data analysis method, computing device and storage medium |
| CN114490744A (en) * | 2021-12-13 | 2022-05-13 | 中企云链（北京）金融信息服务有限公司 | Data caching method, storage medium and electronic device |
| CN114490744B (en) * | 2021-12-13 | 2024-04-26 | 中企云链股份有限公司 | Data caching method, storage medium and electronic device |
| CN114969083A (en) * | 2022-06-24 | 2022-08-30 | 在线途游（北京）科技有限公司 | Real-time data analysis method and system |
| CN115391429A (en) * | 2022-09-14 | 2022-11-25 | 武汉奋进智能机器有限公司 | Time sequence data processing method and device based on big data cloud computing |
| CN115714718A (en) * | 2022-09-23 | 2023-02-24 | 上海芯赛云计算科技有限公司 | Log early warning method and system based on memory, computer equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209258A (en) | Tax end system log real-time analysis method, equipment, medium and system | |
| CN109542733B (en) | High-reliability real-time log collection and visual retrieval method | |
| CN105824744B (en) | A kind of real-time logs capturing analysis method based on B2B platform | |
| CN106815254B (en) | Data processing method and device | |
| CN112506743A (en) | Log monitoring method and device and server | |
| CN108985981B (en) | Data processing system and method | |
| US20160055044A1 (en) | Fault analysis method, fault analysis system, and storage medium | |
| CN113360554A (en) | Method and equipment for extracting, converting and loading ETL (extract transform load) data | |
| WO2015020922A1 (en) | Dynamic collection analysis and reporting of telemetry data | |
| CN106940677A (en) | One kind application daily record data alarm method and device | |
| CN109918349A (en) | Log processing method, device, storage medium and electronic device | |
| CN103729446A (en) | Processing method and device for user operation data and server | |
| CN102567185B (en) | Monitoring method of application server | |
| US10044820B2 (en) | Method and system for automated transaction analysis | |
| CN109885453B (en) | Big data platform monitoring system based on stream data processing | |
| CN107169143B (en) | Efficient mass public opinion data information cluster matching method | |
| CN109167672B (en) | Return source error positioning method, device, storage medium and system | |
| KR20150118963A (en) | Queue monitoring and visualization | |
| CN110888985A (en) | Alarm information processing method and device, electronic equipment and storage medium | |
| CN105069029B (en) | A kind of real-time ETL system and method | |
| US20150281037A1 (en) | Monitoring omission specifying program, monitoring omission specifying method, and monitoring omission specifying device | |
| CN110019152A (en) | A kind of big data cleaning method | |
| Liu et al. | System anomaly detection in distributed systems through MapReduce-Based log analysis | |
| CN113778810A (en) | Log collection method, device and system | |
| CN115914064A (en) | Network system service performance evaluation method, device, computing equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200529 |